2023-04-06 17:56:11,234 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70 2023-04-06 17:56:11,247 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-06 17:56:11,263 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-06 17:56:11,264 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/cluster_00f36afb-ee05-a08a-beea-45e5c08d2a6c, deleteOnExit=true 2023-04-06 17:56:11,264 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-06 17:56:11,265 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/test.cache.data in system properties and HBase conf 2023-04-06 17:56:11,265 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/hadoop.tmp.dir in system properties and HBase conf 2023-04-06 17:56:11,266 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/hadoop.log.dir in system properties and HBase conf 2023-04-06 17:56:11,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-06 17:56:11,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-06 17:56:11,268 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-06 17:56:11,397 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-06 17:56:11,783 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-06 17:56:11,787 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-06 17:56:11,787 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-06 17:56:11,787 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-06 17:56:11,787 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-06 17:56:11,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-06 17:56:11,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-06 17:56:11,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-06 17:56:11,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-06 17:56:11,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-06 17:56:11,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/nfs.dump.dir in system properties and HBase conf 2023-04-06 17:56:11,790 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/java.io.tmpdir in system properties and HBase conf 2023-04-06 17:56:11,790 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-06 17:56:11,790 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-06 17:56:11,791 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-06 17:56:12,270 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-06 17:56:12,273 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-06 17:56:12,543 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-06 17:56:12,722 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-06 17:56:12,737 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-06 17:56:12,773 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-06 17:56:12,805 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/java.io.tmpdir/Jetty_localhost_localdomain_39505_hdfs____.eqjaky/webapp 2023-04-06 17:56:12,941 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:39505 2023-04-06 17:56:12,975 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-06 17:56:12,975 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-06 17:56:13,384 WARN [Listener at localhost.localdomain/35475] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-06 17:56:13,482 WARN [Listener at localhost.localdomain/35475] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-06 17:56:13,503 WARN [Listener at localhost.localdomain/35475] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-06 17:56:13,510 INFO [Listener at localhost.localdomain/35475] log.Slf4jLog(67): jetty-6.1.26 2023-04-06 17:56:13,517 INFO [Listener at localhost.localdomain/35475] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/java.io.tmpdir/Jetty_localhost_36277_datanode____.vhzqkq/webapp 2023-04-06 17:56:13,618 INFO [Listener at localhost.localdomain/35475] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36277 2023-04-06 17:56:13,980 WARN [Listener at localhost.localdomain/40169] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-06 17:56:14,337 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6be60f29feae0f: Processing first storage report for DS-050823ee-9c4a-42d3-9d37-29158d673307 from datanode a158d5e1-6698-4857-9005-71ea66cb6362 2023-04-06 17:56:14,339 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6be60f29feae0f: from storage DS-050823ee-9c4a-42d3-9d37-29158d673307 node DatanodeRegistration(127.0.0.1:42983, datanodeUuid=a158d5e1-6698-4857-9005-71ea66cb6362, infoPort=32853, infoSecurePort=0, ipcPort=40169, storageInfo=lv=-57;cid=testClusterID;nsid=1106585354;c=1680803772339), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-06 17:56:14,339 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6be60f29feae0f: Processing first storage report for DS-ac717d16-2829-43dd-b810-fcf1866fd9a3 from datanode a158d5e1-6698-4857-9005-71ea66cb6362 2023-04-06 17:56:14,339 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6be60f29feae0f: from storage DS-ac717d16-2829-43dd-b810-fcf1866fd9a3 node DatanodeRegistration(127.0.0.1:42983, datanodeUuid=a158d5e1-6698-4857-9005-71ea66cb6362, infoPort=32853, infoSecurePort=0, ipcPort=40169, storageInfo=lv=-57;cid=testClusterID;nsid=1106585354;c=1680803772339), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-06 17:56:14,529 DEBUG [Listener at localhost.localdomain/40169] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70 2023-04-06 17:56:14,584 INFO [Listener at localhost.localdomain/40169] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/cluster_00f36afb-ee05-a08a-beea-45e5c08d2a6c/zookeeper_0, clientPort=49384, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/cluster_00f36afb-ee05-a08a-beea-45e5c08d2a6c/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/cluster_00f36afb-ee05-a08a-beea-45e5c08d2a6c/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-06 17:56:14,594 INFO [Listener at localhost.localdomain/40169] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=49384 2023-04-06 17:56:14,601 INFO [Listener at localhost.localdomain/40169] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 17:56:14,603 INFO [Listener at localhost.localdomain/40169] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 17:56:15,246 INFO [Listener at localhost.localdomain/40169] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741 with version=8 2023-04-06 17:56:15,246 INFO [Listener at localhost.localdomain/40169] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/hbase-staging 2023-04-06 17:56:15,521 INFO [Listener at localhost.localdomain/40169] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-06 17:56:15,936 INFO [Listener at localhost.localdomain/40169] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-06 17:56:15,969 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 17:56:15,969 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-06 17:56:15,969 INFO [Listener at localhost.localdomain/40169] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-06 17:56:15,969 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 17:56:15,970 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-06 17:56:16,118 INFO [Listener at localhost.localdomain/40169] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-06 17:56:16,181 DEBUG [Listener at localhost.localdomain/40169] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-06 17:56:16,257 INFO [Listener at localhost.localdomain/40169] ipc.NettyRpcServer(120): Bind to /148.251.75.209:40911 2023-04-06 17:56:16,265 INFO [Listener at localhost.localdomain/40169] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 17:56:16,267 INFO [Listener at localhost.localdomain/40169] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 17:56:16,284 INFO [Listener at localhost.localdomain/40169] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40911 connecting to ZooKeeper ensemble=127.0.0.1:49384 2023-04-06 17:56:16,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:409110x0, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-06 17:56:16,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40911-0x100876c58980000 connected 2023-04-06 17:56:16,368 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 17:56:16,369 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 17:56:16,372 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-06 17:56:16,378 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40911 2023-04-06 17:56:16,378 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40911 2023-04-06 17:56:16,379 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40911 2023-04-06 17:56:16,379 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40911 2023-04-06 17:56:16,379 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40911 2023-04-06 17:56:16,384 INFO [Listener at localhost.localdomain/40169] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741, hbase.cluster.distributed=false 2023-04-06 17:56:16,440 INFO [Listener at localhost.localdomain/40169] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-06 17:56:16,440 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 17:56:16,440 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-06 17:56:16,440 INFO [Listener at localhost.localdomain/40169] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-06 17:56:16,440 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 17:56:16,441 INFO [Listener at localhost.localdomain/40169] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-06 17:56:16,444 INFO [Listener at localhost.localdomain/40169] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-06 17:56:16,447 INFO [Listener at localhost.localdomain/40169] ipc.NettyRpcServer(120): Bind to /148.251.75.209:38773 2023-04-06 17:56:16,449 INFO [Listener at localhost.localdomain/40169] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-06 17:56:16,453 DEBUG [Listener at localhost.localdomain/40169] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-06 17:56:16,454 INFO [Listener at localhost.localdomain/40169] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 17:56:16,457 INFO [Listener at localhost.localdomain/40169] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 17:56:16,458 INFO [Listener at localhost.localdomain/40169] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38773 connecting to ZooKeeper ensemble=127.0.0.1:49384 2023-04-06 17:56:16,462 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:387730x0, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-06 17:56:16,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38773-0x100876c58980001 connected 2023-04-06 17:56:16,463 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 17:56:16,464 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 17:56:16,465 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-06 17:56:16,465 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38773 2023-04-06 17:56:16,465 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38773 2023-04-06 17:56:16,470 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38773 2023-04-06 17:56:16,470 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38773 2023-04-06 17:56:16,471 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38773 2023-04-06 17:56:16,475 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:16,482 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-06 17:56:16,483 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:16,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-06 17:56:16,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-06 17:56:16,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:16,499 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-06 17:56:16,500 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,40911,1680803775368 from backup master directory 2023-04-06 17:56:16,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-06 17:56:16,502 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:16,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-06 17:56:16,503 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-06 17:56:16,503 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:16,505 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-06 17:56:16,506 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-06 17:56:16,979 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/hbase.id with ID: 2af07e11-1128-4974-b133-caa2e41a8816 2023-04-06 17:56:17,035 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 17:56:17,054 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:17,103 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2a2c2b34 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:17,138 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7acf060a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:17,166 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:56:17,168 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-06 17:56:17,186 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-06 17:56:17,186 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-06 17:56:17,188 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:17,191 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:17,193 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 17:56:17,226 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store-tmp 2023-04-06 17:56:17,258 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:17,258 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-06 17:56:17,259 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:56:17,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:56:17,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-06 17:56:17,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:56:17,259 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:56:17,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 17:56:17,261 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/WALs/jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:17,288 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C40911%2C1680803775368, suffix=, logDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/WALs/jenkins-hbase20.apache.org,40911,1680803775368, archiveDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/oldWALs, maxLogs=10 2023-04-06 17:56:17,355 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42983,DS-050823ee-9c4a-42d3-9d37-29158d673307,DISK] 2023-04-06 17:56:17,364 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:17,438 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/WALs/jenkins-hbase20.apache.org,40911,1680803775368/jenkins-hbase20.apache.org%2C40911%2C1680803775368.1680803777302 2023-04-06 17:56:17,439 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42983,DS-050823ee-9c4a-42d3-9d37-29158d673307,DISK]] 2023-04-06 17:56:17,440 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:56:17,440 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:17,443 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 17:56:17,445 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 17:56:17,510 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-06 17:56:17,517 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-06 17:56:17,537 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-06 17:56:17,549 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:17,555 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-06 17:56:17,557 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-06 17:56:17,573 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 17:56:17,578 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:56:17,579 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58763874, jitterRate=-0.12435004115104675}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-06 17:56:17,580 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 17:56:17,581 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-06 17:56:17,603 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-06 17:56:17,604 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-06 17:56:17,606 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-06 17:56:17,608 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-06 17:56:17,638 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 29 msec 2023-04-06 17:56:17,638 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-06 17:56:17,664 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-06 17:56:17,670 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-06 17:56:17,693 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-06 17:56:17,696 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-06 17:56:17,698 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-06 17:56:17,704 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-06 17:56:17,707 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-06 17:56:17,710 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:17,711 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-06 17:56:17,712 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-06 17:56:17,722 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-06 17:56:17,726 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-06 17:56:17,726 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-06 17:56:17,726 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:17,726 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,40911,1680803775368, sessionid=0x100876c58980000, setting cluster-up flag (Was=false) 2023-04-06 17:56:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:17,745 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-06 17:56:17,747 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:17,752 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:17,756 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-06 17:56:17,757 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:17,760 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.hbase-snapshot/.tmp 2023-04-06 17:56:17,777 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(952): ClusterId : 2af07e11-1128-4974-b133-caa2e41a8816 2023-04-06 17:56:17,782 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-06 17:56:17,786 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-06 17:56:17,786 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-06 17:56:17,789 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-06 17:56:17,789 DEBUG [RS:0;jenkins-hbase20:38773] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3b6661 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:17,801 DEBUG [RS:0;jenkins-hbase20:38773] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10fc9bd6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:17,802 DEBUG [RS:0;jenkins-hbase20:38773] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14c30d11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-06 17:56:17,829 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:38773 2023-04-06 17:56:17,834 INFO [RS:0;jenkins-hbase20:38773] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-06 17:56:17,834 INFO [RS:0;jenkins-hbase20:38773] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-06 17:56:17,834 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1023): About to register with Master. 2023-04-06 17:56:17,837 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,40911,1680803775368 with isa=jenkins-hbase20.apache.org/148.251.75.209:38773, startcode=1680803776439 2023-04-06 17:56:17,852 DEBUG [RS:0;jenkins-hbase20:38773] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-06 17:56:17,911 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-06 17:56:17,916 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37557, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-06 17:56:17,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 17:56:17,920 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 17:56:17,920 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 17:56:17,920 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 17:56:17,920 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-06 17:56:17,920 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:17,920 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-06 17:56:17,920 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:17,928 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680803807928 2023-04-06 17:56:17,930 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-06 17:56:17,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:17,939 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-06 17:56:17,940 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-06 17:56:17,946 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-06 17:56:17,948 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-06 17:56:17,954 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-06 17:56:17,955 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-06 17:56:17,956 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-06 17:56:17,956 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-06 17:56:17,960 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:17,963 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-06 17:56:17,964 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-06 17:56:17,964 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-06 17:56:17,967 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-06 17:56:17,968 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-06 17:56:17,969 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(2829): Master is not running yet 2023-04-06 17:56:17,970 WARN [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-06 17:56:17,970 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680803777970,5,FailOnTimeoutGroup] 2023-04-06 17:56:17,970 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680803777970,5,FailOnTimeoutGroup] 2023-04-06 17:56:17,971 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:17,971 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-06 17:56:17,972 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:17,972 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:18,071 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,40911,1680803775368 with isa=jenkins-hbase20.apache.org/148.251.75.209:38773, startcode=1680803776439 2023-04-06 17:56:18,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:18,087 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741 2023-04-06 17:56:18,087 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:35475 2023-04-06 17:56:18,087 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-06 17:56:18,094 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 17:56:18,095 DEBUG [RS:0;jenkins-hbase20:38773] zookeeper.ZKUtil(162): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:18,095 WARN [RS:0;jenkins-hbase20:38773] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-06 17:56:18,096 INFO [RS:0;jenkins-hbase20:38773] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 17:56:18,097 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/WALs/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:18,098 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,38773,1680803776439] 2023-04-06 17:56:18,111 DEBUG [RS:0;jenkins-hbase20:38773] zookeeper.ZKUtil(162): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:18,124 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-06 17:56:18,136 INFO [RS:0;jenkins-hbase20:38773] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-06 17:56:18,162 INFO [RS:0;jenkins-hbase20:38773] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-06 17:56:18,166 INFO [RS:0;jenkins-hbase20:38773] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-06 17:56:18,166 INFO [RS:0;jenkins-hbase20:38773] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:18,167 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-06 17:56:18,173 INFO [RS:0;jenkins-hbase20:38773] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:18,173 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,174 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,174 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,174 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,174 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,174 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-06 17:56:18,175 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,175 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,175 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,175 DEBUG [RS:0;jenkins-hbase20:38773] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 17:56:18,176 INFO [RS:0;jenkins-hbase20:38773] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:18,176 INFO [RS:0;jenkins-hbase20:38773] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:18,176 INFO [RS:0;jenkins-hbase20:38773] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:18,190 INFO [RS:0;jenkins-hbase20:38773] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-06 17:56:18,192 INFO [RS:0;jenkins-hbase20:38773] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38773,1680803776439-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:18,211 INFO [RS:0;jenkins-hbase20:38773] regionserver.Replication(203): jenkins-hbase20.apache.org,38773,1680803776439 started 2023-04-06 17:56:18,211 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,38773,1680803776439, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:38773, sessionid=0x100876c58980001 2023-04-06 17:56:18,212 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-06 17:56:18,212 DEBUG [RS:0;jenkins-hbase20:38773] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:18,212 DEBUG [RS:0;jenkins-hbase20:38773] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,38773,1680803776439' 2023-04-06 17:56:18,212 DEBUG [RS:0;jenkins-hbase20:38773] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:18,213 DEBUG [RS:0;jenkins-hbase20:38773] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:18,213 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-06 17:56:18,213 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-06 17:56:18,214 DEBUG [RS:0;jenkins-hbase20:38773] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:18,214 DEBUG [RS:0;jenkins-hbase20:38773] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,38773,1680803776439' 2023-04-06 17:56:18,214 DEBUG [RS:0;jenkins-hbase20:38773] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-06 17:56:18,214 DEBUG [RS:0;jenkins-hbase20:38773] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-06 17:56:18,215 DEBUG [RS:0;jenkins-hbase20:38773] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-06 17:56:18,215 INFO [RS:0;jenkins-hbase20:38773] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-06 17:56:18,215 INFO [RS:0;jenkins-hbase20:38773] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-06 17:56:18,325 INFO [RS:0;jenkins-hbase20:38773] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38773%2C1680803776439, suffix=, logDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/WALs/jenkins-hbase20.apache.org,38773,1680803776439, archiveDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/oldWALs, maxLogs=32 2023-04-06 17:56:18,343 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42983,DS-050823ee-9c4a-42d3-9d37-29158d673307,DISK] 2023-04-06 17:56:18,347 INFO [RS:0;jenkins-hbase20:38773] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/WALs/jenkins-hbase20.apache.org,38773,1680803776439/jenkins-hbase20.apache.org%2C38773%2C1680803776439.1680803778328 2023-04-06 17:56:18,347 DEBUG [RS:0;jenkins-hbase20:38773] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42983,DS-050823ee-9c4a-42d3-9d37-29158d673307,DISK]] 2023-04-06 17:56:18,388 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-06 17:56:18,389 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-06 17:56:18,389 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741 2023-04-06 17:56:18,805 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:18,808 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-06 17:56:18,811 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/info 2023-04-06 17:56:18,811 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-06 17:56:18,813 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:18,813 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-06 17:56:18,815 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/rep_barrier 2023-04-06 17:56:18,816 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-06 17:56:18,816 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:18,817 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-06 17:56:18,819 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/table 2023-04-06 17:56:18,819 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-06 17:56:18,820 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:18,822 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740 2023-04-06 17:56:18,823 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740 2023-04-06 17:56:18,826 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:56:18,828 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-06 17:56:18,831 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:56:18,833 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72752184, jitterRate=0.08409202098846436}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:56:18,833 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-06 17:56:18,833 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-06 17:56:18,833 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-06 17:56:18,833 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-06 17:56:18,833 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-06 17:56:18,833 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-06 17:56:18,834 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-06 17:56:18,834 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-06 17:56:18,839 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-06 17:56:18,839 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-06 17:56:18,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-06 17:56:18,859 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-06 17:56:18,862 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-06 17:56:19,014 DEBUG [jenkins-hbase20:40911] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-06 17:56:19,017 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,38773,1680803776439, state=OPENING 2023-04-06 17:56:19,024 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-06 17:56:19,025 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:19,026 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-06 17:56:19,028 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:56:19,213 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:19,216 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-06 17:56:19,220 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-06 17:56:19,231 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-06 17:56:19,232 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 17:56:19,235 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38773%2C1680803776439.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/WALs/jenkins-hbase20.apache.org,38773,1680803776439, archiveDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/oldWALs, maxLogs=32 2023-04-06 17:56:19,251 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42983,DS-050823ee-9c4a-42d3-9d37-29158d673307,DISK] 2023-04-06 17:56:19,255 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/WALs/jenkins-hbase20.apache.org,38773,1680803776439/jenkins-hbase20.apache.org%2C38773%2C1680803776439.meta.1680803779237.meta 2023-04-06 17:56:19,256 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42983,DS-050823ee-9c4a-42d3-9d37-29158d673307,DISK]] 2023-04-06 17:56:19,256 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:56:19,258 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-06 17:56:19,272 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-06 17:56:19,276 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-06 17:56:19,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-06 17:56:19,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:19,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-06 17:56:19,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-06 17:56:19,285 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-06 17:56:19,286 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/info 2023-04-06 17:56:19,287 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/info 2023-04-06 17:56:19,288 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-06 17:56:19,289 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:19,289 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-06 17:56:19,290 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/rep_barrier 2023-04-06 17:56:19,290 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/rep_barrier 2023-04-06 17:56:19,291 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-06 17:56:19,292 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:19,292 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-06 17:56:19,293 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/table 2023-04-06 17:56:19,293 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/table 2023-04-06 17:56:19,294 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-06 17:56:19,295 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:19,296 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740 2023-04-06 17:56:19,299 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740 2023-04-06 17:56:19,302 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:56:19,304 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-06 17:56:19,306 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61274024, jitterRate=-0.08694589138031006}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:56:19,306 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-06 17:56:19,314 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680803779204 2023-04-06 17:56:19,333 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-06 17:56:19,333 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-06 17:56:19,334 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,38773,1680803776439, state=OPEN 2023-04-06 17:56:19,336 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-06 17:56:19,336 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-06 17:56:19,341 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-06 17:56:19,341 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,38773,1680803776439 in 308 msec 2023-04-06 17:56:19,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-06 17:56:19,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 496 msec 2023-04-06 17:56:19,351 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5250 sec 2023-04-06 17:56:19,351 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680803779351, completionTime=-1 2023-04-06 17:56:19,352 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-06 17:56:19,352 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-06 17:56:19,416 DEBUG [hconnection-0x2ef4de32-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:19,420 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:19,452 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-06 17:56:19,453 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680803839453 2023-04-06 17:56:19,453 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680803899453 2023-04-06 17:56:19,453 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 101 msec 2023-04-06 17:56:19,476 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40911,1680803775368-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:19,477 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40911,1680803775368-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:19,477 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40911,1680803775368-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:19,478 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:40911, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:19,479 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-06 17:56:19,485 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-06 17:56:19,492 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-06 17:56:19,494 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-06 17:56:19,503 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-06 17:56:19,507 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 17:56:19,510 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 17:56:19,533 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:19,535 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e empty. 2023-04-06 17:56:19,536 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:19,536 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-06 17:56:19,960 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-06 17:56:19,964 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0d49654bd69ee321ab98d34d9a593b8e, NAME => 'hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp 2023-04-06 17:56:20,388 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:20,388 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 0d49654bd69ee321ab98d34d9a593b8e, disabling compactions & flushes 2023-04-06 17:56:20,389 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:56:20,389 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:56:20,389 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. after waiting 0 ms 2023-04-06 17:56:20,389 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:56:20,389 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:56:20,390 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 0d49654bd69ee321ab98d34d9a593b8e: 2023-04-06 17:56:20,397 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 17:56:20,416 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680803780401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803780401"}]},"ts":"1680803780401"} 2023-04-06 17:56:20,438 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 17:56:20,440 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 17:56:20,444 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803780441"}]},"ts":"1680803780441"} 2023-04-06 17:56:20,448 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-06 17:56:20,456 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=0d49654bd69ee321ab98d34d9a593b8e, ASSIGN}] 2023-04-06 17:56:20,459 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=0d49654bd69ee321ab98d34d9a593b8e, ASSIGN 2023-04-06 17:56:20,462 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=0d49654bd69ee321ab98d34d9a593b8e, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=false 2023-04-06 17:56:20,613 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=0d49654bd69ee321ab98d34d9a593b8e, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:20,614 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680803780613"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803780613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803780613"}]},"ts":"1680803780613"} 2023-04-06 17:56:20,627 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 0d49654bd69ee321ab98d34d9a593b8e, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:56:20,793 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:56:20,793 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0d49654bd69ee321ab98d34d9a593b8e, NAME => 'hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:56:20,795 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:20,795 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:20,796 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:20,796 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:20,799 INFO [StoreOpener-0d49654bd69ee321ab98d34d9a593b8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:20,803 DEBUG [StoreOpener-0d49654bd69ee321ab98d34d9a593b8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/info 2023-04-06 17:56:20,803 DEBUG [StoreOpener-0d49654bd69ee321ab98d34d9a593b8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/info 2023-04-06 17:56:20,803 INFO [StoreOpener-0d49654bd69ee321ab98d34d9a593b8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0d49654bd69ee321ab98d34d9a593b8e columnFamilyName info 2023-04-06 17:56:20,804 INFO [StoreOpener-0d49654bd69ee321ab98d34d9a593b8e-1] regionserver.HStore(310): Store=0d49654bd69ee321ab98d34d9a593b8e/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:20,807 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:20,808 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:20,813 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:56:20,816 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:56:20,817 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 0d49654bd69ee321ab98d34d9a593b8e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66679565, jitterRate=-0.006397053599357605}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-06 17:56:20,817 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 0d49654bd69ee321ab98d34d9a593b8e: 2023-04-06 17:56:20,819 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e., pid=6, masterSystemTime=1680803780781 2023-04-06 17:56:20,822 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:56:20,822 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:56:20,824 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=0d49654bd69ee321ab98d34d9a593b8e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:20,824 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680803780823"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803780823"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803780823"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803780823"}]},"ts":"1680803780823"} 2023-04-06 17:56:20,831 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-06 17:56:20,831 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 0d49654bd69ee321ab98d34d9a593b8e, server=jenkins-hbase20.apache.org,38773,1680803776439 in 201 msec 2023-04-06 17:56:20,833 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-06 17:56:20,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=0d49654bd69ee321ab98d34d9a593b8e, ASSIGN in 375 msec 2023-04-06 17:56:20,835 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 17:56:20,835 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803780835"}]},"ts":"1680803780835"} 2023-04-06 17:56:20,839 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-06 17:56:20,842 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 17:56:20,845 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3470 sec 2023-04-06 17:56:20,907 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-06 17:56:20,909 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-06 17:56:20,909 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:56:20,950 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-06 17:56:20,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-06 17:56:20,971 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-04-06 17:56:20,983 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-06 17:56:20,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-06 17:56:20,999 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-06 17:56:21,008 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-06 17:56:21,009 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-06 17:56:21,010 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.507sec 2023-04-06 17:56:21,012 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-06 17:56:21,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-06 17:56:21,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-06 17:56:21,016 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40911,1680803775368-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-06 17:56:21,016 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40911,1680803775368-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-06 17:56:21,028 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-06 17:56:21,092 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3eb02786 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:21,099 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@975c493, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:21,111 DEBUG [hconnection-0x440e158d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:21,124 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55744, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:21,132 INFO [Listener at localhost.localdomain/40169] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:56:21,151 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=181, ProcessCount=168, AvailableMemoryMB=3582 2023-04-06 17:56:21,166 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 17:56:21,169 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 17:56:21,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:56:21,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:56:21,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:21,186 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 17:56:21,188 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 17:56:21,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-06 17:56:21,191 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,192 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798 empty. 2023-04-06 17:56:21,193 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,194 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:56:21,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 17:56:21,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 17:56:21,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 17:56:21,612 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:56:21,617 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1a4cbf1c6f3aeda9a5b63ebd9e930798, NAME => 'TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp 2023-04-06 17:56:21,629 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:21,629 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1a4cbf1c6f3aeda9a5b63ebd9e930798, disabling compactions & flushes 2023-04-06 17:56:21,629 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:21,630 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:21,630 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. after waiting 0 ms 2023-04-06 17:56:21,630 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:21,630 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:21,630 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:21,634 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 17:56:21,636 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803781636"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803781636"}]},"ts":"1680803781636"} 2023-04-06 17:56:21,639 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 17:56:21,641 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 17:56:21,641 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803781641"}]},"ts":"1680803781641"} 2023-04-06 17:56:21,643 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 17:56:21,646 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a4cbf1c6f3aeda9a5b63ebd9e930798, ASSIGN}] 2023-04-06 17:56:21,648 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a4cbf1c6f3aeda9a5b63ebd9e930798, ASSIGN 2023-04-06 17:56:21,649 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a4cbf1c6f3aeda9a5b63ebd9e930798, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=false 2023-04-06 17:56:21,801 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1a4cbf1c6f3aeda9a5b63ebd9e930798, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:21,802 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803781801"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803781801"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803781801"}]},"ts":"1680803781801"} 2023-04-06 17:56:21,808 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:56:21,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 17:56:21,970 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:21,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1a4cbf1c6f3aeda9a5b63ebd9e930798, NAME => 'TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:56:21,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:21,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,974 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,977 DEBUG [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A 2023-04-06 17:56:21,977 DEBUG [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A 2023-04-06 17:56:21,979 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:21,980 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4cbf1c6f3aeda9a5b63ebd9e930798 columnFamilyName A 2023-04-06 17:56:21,981 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.HStore(310): Store=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:21,981 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,984 DEBUG [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B 2023-04-06 17:56:21,984 DEBUG [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B 2023-04-06 17:56:21,985 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:21,985 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4cbf1c6f3aeda9a5b63ebd9e930798 columnFamilyName B 2023-04-06 17:56:21,986 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.HStore(310): Store=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:21,986 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,988 DEBUG [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C 2023-04-06 17:56:21,988 DEBUG [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C 2023-04-06 17:56:21,988 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:21,989 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a4cbf1c6f3aeda9a5b63ebd9e930798 columnFamilyName C 2023-04-06 17:56:21,989 INFO [StoreOpener-1a4cbf1c6f3aeda9a5b63ebd9e930798-1] regionserver.HStore(310): Store=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:21,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:21,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:21,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:56:21,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:22,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:56:22,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1a4cbf1c6f3aeda9a5b63ebd9e930798; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67585407, jitterRate=0.007101044058799744}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:56:22,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:22,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., pid=11, masterSystemTime=1680803781963 2023-04-06 17:56:22,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:22,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:22,010 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1a4cbf1c6f3aeda9a5b63ebd9e930798, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,010 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803782010"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803782010"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803782010"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803782010"}]},"ts":"1680803782010"} 2023-04-06 17:56:22,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-06 17:56:22,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 in 205 msec 2023-04-06 17:56:22,020 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-06 17:56:22,021 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a4cbf1c6f3aeda9a5b63ebd9e930798, ASSIGN in 371 msec 2023-04-06 17:56:22,022 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 17:56:22,022 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803782022"}]},"ts":"1680803782022"} 2023-04-06 17:56:22,024 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 17:56:22,027 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 17:56:22,029 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 845 msec 2023-04-06 17:56:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 17:56:22,316 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-06 17:56:22,326 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x41908139 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,330 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c040f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,333 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,337 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,340 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 17:56:22,343 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 17:56:22,351 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x37b22714 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,356 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d07aaa4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,357 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x78c6cbc8 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,366 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e781aa0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,367 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x754370ad to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,373 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d6b0f87, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,374 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3339866b to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,378 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@782365ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,378 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x6194be61 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,382 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3627406c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,383 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x0e18c51d to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,387 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7de00027, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,388 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x31222f6c to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,419 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@795a2478, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,420 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x09048085 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,434 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21551c39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,435 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x57966649 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:22,442 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abbc496, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:22,449 DEBUG [hconnection-0x1878412-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,449 DEBUG [hconnection-0x7cbe57fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,450 DEBUG [hconnection-0x67b13574-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,454 DEBUG [hconnection-0x7e1f6218-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,454 DEBUG [hconnection-0x6adbc389-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,455 DEBUG [hconnection-0x3422d0cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,460 DEBUG [hconnection-0x7e1b0869-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,460 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55774, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,461 DEBUG [hconnection-0x39474c17-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,461 DEBUG [hconnection-0x709f9e55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:22,462 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,462 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,465 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,466 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,465 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,466 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,467 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55844, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,471 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55856, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:22,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:22,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:22,523 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:22,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:22,524 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:22,524 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:22,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:22,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:22,527 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:22,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:22,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:22,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:22,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:22,555 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:22,556 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:22,558 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:22,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:22,563 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:22,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:22,566 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:22,568 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 17:56:22,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:22,591 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:22,599 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:22,600 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:22,600 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:22,600 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:22,600 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:22,600 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803842721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803842721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803782663, max timestamp=1680803782721Num uniques 859; 2023-04-06 17:56:22,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:56:22,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803782663, max timestamp=1680803782722Num uniques 859; 2023-04-06 17:56:22,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:56:22,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803782663, max timestamp=1680803782720Num uniques 859; 2023-04-06 17:56:22,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:56:22,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803842775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803842779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803842786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803842844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803842844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803842889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803842892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:22,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:22,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803842892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803843052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803843050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,093 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/bad127d70e6a425cb33713c6edb19eac 2023-04-06 17:56:23,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803843098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803843099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803843099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,192 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/7683ad0ead04480aa7eb1c4a67f2e810 2023-04-06 17:56:23,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803843355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803843355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803843406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803843406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803843407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,645 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/50c61f132c364f698542b837fb803130 2023-04-06 17:56:23,655 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/bad127d70e6a425cb33713c6edb19eac as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/bad127d70e6a425cb33713c6edb19eac 2023-04-06 17:56:23,665 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/bad127d70e6a425cb33713c6edb19eac, entries=150, sequenceid=12, filesize=11.6 K 2023-04-06 17:56:23,671 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/7683ad0ead04480aa7eb1c4a67f2e810 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7683ad0ead04480aa7eb1c4a67f2e810 2023-04-06 17:56:23,690 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7683ad0ead04480aa7eb1c4a67f2e810, entries=150, sequenceid=12, filesize=11.6 K 2023-04-06 17:56:23,694 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/50c61f132c364f698542b837fb803130 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/50c61f132c364f698542b837fb803130 2023-04-06 17:56:23,713 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/50c61f132c364f698542b837fb803130, entries=150, sequenceid=12, filesize=11.6 K 2023-04-06 17:56:23,716 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.28 KB/129315 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 1148ms, sequenceid=12, compaction requested=false 2023-04-06 17:56:23,719 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-06 17:56:23,722 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:23,722 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:23,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:23,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:23,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:23,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:23,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:23,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:23,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f88a12a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:23,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,741 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,741 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:23,741 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:23,741 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:23,741 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:23,744 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,744 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:23,744 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:23,744 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:23,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:23,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:23,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:23,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:23,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:23,759 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,759 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,759 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:23,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:23,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:23,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:23,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:23,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:23,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:23,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:23,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:23,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:23,775 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:23,782 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:23,782 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:23,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:23,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:23,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=132.99 KB heapSize=382.24 KB 2023-04-06 17:56:23,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:23,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:23,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:23,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:23,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:23,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:23,882 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:23,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:23,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:23,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:23,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:23,932 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:23,932 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:23,933 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:23,934 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:23,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/2170e9306a004cf4beb8e9e577fff771 2023-04-06 17:56:23,934 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,934 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:23,936 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,936 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:23,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:23,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:23,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:23,938 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,939 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:23,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803843926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:23,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:23,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:23,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:23,958 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:23,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:23,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:23,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:23,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:23,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:23,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:23,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:23,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:23,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:23,966 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803843936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,969 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,969 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:23,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:23,971 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,971 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803843940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:23,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:23,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e27db2e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:23,974 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:23,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803843941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:23,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803843946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,982 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:23,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:23,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:23,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:23,982 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:23,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:23,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:23,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:23,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:24,003 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:24,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:24,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:24,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:24,004 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:24,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,017 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,026 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,026 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:24,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:24,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,031 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:24,031 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803844068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803844075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803844078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803844080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,130 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-06 17:56:24,133 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-06 17:56:24,134 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:24,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:24,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:24,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:24,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:24,146 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:24,146 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:24,151 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:24,151 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,151 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,153 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,155 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,155 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:24,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:24,156 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:24,156 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:24,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:24,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:24,157 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,157 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:24,157 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:24,157 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:24,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:24,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,159 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:24,159 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:24,165 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,165 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,166 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,166 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ddc5faa[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:24,168 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:24,168 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,172 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:24,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:24,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:24,183 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:24,193 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:24,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:24,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:24,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:24,193 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:24,195 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,195 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,195 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,201 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:24,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:24,211 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:24,210 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,211 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:24,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803844287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803844287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803844289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803844291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,318 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:24,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:24,330 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:24,330 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:24,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:24,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:24,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:24,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:24,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:24,336 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:24,336 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,339 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:24,340 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:24,340 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:24,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,343 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:24,343 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:24,343 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:24,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,350 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:24,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b12a9e5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:24,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:24,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:24,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:24,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:24,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:24,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:24,362 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:24,362 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:24,362 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:24,362 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:24,363 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:24,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,386 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,386 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,386 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,386 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,387 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:24,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:24,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:24,391 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:24,391 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:24,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/315ed765b92448e4accc1ad36328b669 2023-04-06 17:56:24,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/6cb67ab7566e4d789be2d3b717c5a719 2023-04-06 17:56:24,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/2170e9306a004cf4beb8e9e577fff771 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2170e9306a004cf4beb8e9e577fff771 2023-04-06 17:56:24,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:24,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:24,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:24,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2170e9306a004cf4beb8e9e577fff771, entries=200, sequenceid=49, filesize=13.9 K 2023-04-06 17:56:24,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:24,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:24,506 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:24,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/315ed765b92448e4accc1ad36328b669 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/315ed765b92448e4accc1ad36328b669 2023-04-06 17:56:24,510 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:24,510 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:24,511 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,511 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,518 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,519 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,519 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:24,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:24,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:24,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:24,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:24,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:24,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:24,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:24,526 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,526 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:24,526 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:24,526 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:24,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f9da5c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:24,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:24,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,538 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:24,539 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:24,539 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:24,539 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,539 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:24,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:24,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/315ed765b92448e4accc1ad36328b669, entries=150, sequenceid=49, filesize=11.6 K 2023-04-06 17:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/6cb67ab7566e4d789be2d3b717c5a719 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6cb67ab7566e4d789be2d3b717c5a719 2023-04-06 17:56:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:24,558 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:24,558 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:24,558 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:24,558 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:24,558 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:24,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6cb67ab7566e4d789be2d3b717c5a719, entries=150, sequenceid=49, filesize=11.6 K 2023-04-06 17:56:24,562 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,562 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,562 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,563 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,563 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~132.99 KB/136185, heapSize ~382.20 KB/391368, currentSize=46.96 KB/48090 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 693ms, sequenceid=49, compaction requested=false 2023-04-06 17:56:24,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,564 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:24,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:24,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,575 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,575 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,575 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,575 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:24,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:24,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,575 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,576 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:24,576 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:24,600 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:56:24,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:24,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:24,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803844658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803844658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803844667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803844668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,676 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:24,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:24,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:24,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:24,699 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:24,699 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:24,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:24,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,705 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,705 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,705 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,705 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:24,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:24,707 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:24,707 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,708 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:24,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:24,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:24,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:24,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:24,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:24,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:24,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413781e6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:24,727 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:24,727 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:24,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:24,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:24,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:24,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:24,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:24,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:24,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:24,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:24,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:24,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:24,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:24,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:24,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,756 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,764 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,764 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,764 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,764 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:24,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:24,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,772 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,772 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,772 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,772 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,772 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,772 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:24,773 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:24,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803844774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803844774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803844775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803844775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,851 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 17:56:24,873 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:24,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:24,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:24,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:24,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:24,890 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:24,890 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,891 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:24,891 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:24,892 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,892 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,893 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:24,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:24,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:24,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:24,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:24,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:24,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:24,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:24,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:24,913 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,913 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:24,913 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:24,913 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:24,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:24,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:24,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:24,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:24,918 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:24,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ec9a765[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:24,920 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:24,920 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,922 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:24,922 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:24,922 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:24,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:24,928 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:24,938 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:24,938 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:24,938 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:24,939 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:24,939 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:24,940 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,940 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,940 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:24,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:24,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,945 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,946 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803844952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:24,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:24,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:24,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:24,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:24,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:24,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,959 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:24,959 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:24,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:24,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:24,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:24,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:24,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803844985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803844985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803844988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:24,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:24,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803844993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,059 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:25,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:25,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:25,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:25,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:25,068 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:25,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:25,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/c98794e6936e4e5792ece7bd3c6f9654 2023-04-06 17:56:25,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:25,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:25,078 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,078 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,080 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:25,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:25,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:25,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:25,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:25,095 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,095 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:25,095 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:25,095 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:25,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:25,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:25,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:25,101 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,101 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@101a68d8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:25,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:25,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:25,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:25,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:25,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:25,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:25,117 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:25,122 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:25,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:25,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:25,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:25,123 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:25,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:25,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,135 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:25,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:25,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:25,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,143 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:25,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:25,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:25,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,143 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:25,143 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:25,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:25,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:25,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/300d65636d1c462ea172d0ed9a8c2bfa 2023-04-06 17:56:25,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/565a37fcfc7745c8a4002ed083c004ec 2023-04-06 17:56:25,243 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:25,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/c98794e6936e4e5792ece7bd3c6f9654 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c98794e6936e4e5792ece7bd3c6f9654 2023-04-06 17:56:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:25,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:25,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:25,266 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:25,266 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:25,275 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c98794e6936e4e5792ece7bd3c6f9654, entries=200, sequenceid=61, filesize=13.9 K 2023-04-06 17:56:25,276 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:25,276 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:25,277 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,277 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/300d65636d1c462ea172d0ed9a8c2bfa as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/300d65636d1c462ea172d0ed9a8c2bfa 2023-04-06 17:56:25,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:25,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:25,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,281 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,281 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:25,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:25,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:25,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:25,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:25,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:25,283 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:25,283 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:25,283 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,283 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:25,283 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:25,284 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:25,284 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:25,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:25,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:25,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:25,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/300d65636d1c462ea172d0ed9a8c2bfa, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 17:56:25,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/565a37fcfc7745c8a4002ed083c004ec as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/565a37fcfc7745c8a4002ed083c004ec 2023-04-06 17:56:25,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803845293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:25,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:25,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803845294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:25,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803845294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:25,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803845296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8e7bbd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:25,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:25,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,337 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,337 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,338 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:25,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:25,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:25,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:25,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/565a37fcfc7745c8a4002ed083c004ec, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 17:56:25,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 744ms, sequenceid=61, compaction requested=true 2023-04-06 17:56:25,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:25,348 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:25,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:25,352 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:25,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:25,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:25,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:25,352 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:25,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:25,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:25,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:25,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:25,362 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:25,364 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:25,365 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:25,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:25,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:25,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:25,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:25,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:25,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:25,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:25,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:25,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:25,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:25,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:25,379 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:25,379 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:25,381 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:25,381 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:25,381 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,381 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,381 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7683ad0ead04480aa7eb1c4a67f2e810, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/315ed765b92448e4accc1ad36328b669, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/300d65636d1c462ea172d0ed9a8c2bfa] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=34.8 K 2023-04-06 17:56:25,381 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/bad127d70e6a425cb33713c6edb19eac, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2170e9306a004cf4beb8e9e577fff771, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c98794e6936e4e5792ece7bd3c6f9654] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=39.4 K 2023-04-06 17:56:25,383 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7683ad0ead04480aa7eb1c4a67f2e810, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680803782509 2023-04-06 17:56:25,384 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting bad127d70e6a425cb33713c6edb19eac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680803782509 2023-04-06 17:56:25,391 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 315ed765b92448e4accc1ad36328b669, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680803782769 2023-04-06 17:56:25,393 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 300d65636d1c462ea172d0ed9a8c2bfa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803783936 2023-04-06 17:56:25,394 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 2170e9306a004cf4beb8e9e577fff771, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680803782769 2023-04-06 17:56:25,401 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c98794e6936e4e5792ece7bd3c6f9654, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803783936 2023-04-06 17:56:25,434 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#9 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:25,461 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#10 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:25,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:25,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:25,517 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-06 17:56:25,517 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-06 17:56:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:25,526 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:25,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:25,533 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:25,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:25,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,552 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:25,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:25,558 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,559 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,566 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:25,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:25,580 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/c1206061ff9a4dd19f7d1a86450146b1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c1206061ff9a4dd19f7d1a86450146b1 2023-04-06 17:56:25,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:25,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:25,588 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:25,602 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:25,605 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:56:25,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:25,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:25,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:25,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:25,606 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:25,606 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:25,614 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into c1206061ff9a4dd19f7d1a86450146b1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:25,614 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:25,614 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=13, startTime=1680803785362; duration=0sec 2023-04-06 17:56:25,615 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:25,615 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:25,619 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:25,620 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:25,620 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:25,620 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/50c61f132c364f698542b837fb803130, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6cb67ab7566e4d789be2d3b717c5a719, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/565a37fcfc7745c8a4002ed083c004ec] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=34.8 K 2023-04-06 17:56:25,621 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 50c61f132c364f698542b837fb803130, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680803782509 2023-04-06 17:56:25,622 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6cb67ab7566e4d789be2d3b717c5a719, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680803782769 2023-04-06 17:56:25,622 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 565a37fcfc7745c8a4002ed083c004ec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803783936 2023-04-06 17:56:25,649 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#12 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:25,714 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/7ce36887642d4ff099451fa934d9af0e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ce36887642d4ff099451fa934d9af0e 2023-04-06 17:56:25,753 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 7ce36887642d4ff099451fa934d9af0e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:25,754 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:25,754 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=13, startTime=1680803785362; duration=0sec 2023-04-06 17:56:25,754 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:25,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:25,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803845850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803845852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803845859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803845859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803845962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803845961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,964 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/b0df11ab894b4f1895696367df44136b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0df11ab894b4f1895696367df44136b 2023-04-06 17:56:25,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803845965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:25,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803845966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:25,978 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into b0df11ab894b4f1895696367df44136b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:25,979 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:25,979 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=13, startTime=1680803785349; duration=0sec 2023-04-06 17:56:25,979 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:26,092 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/6925c51ca2854f3e83275bfcfe128d73 2023-04-06 17:56:26,155 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/8777265bbabb4980903bf7b64f33f0c1 2023-04-06 17:56:26,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803846165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803846166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803846169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803846169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,251 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/b2f60c39442348d19b7c2ac181ef0259 2023-04-06 17:56:26,261 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/6925c51ca2854f3e83275bfcfe128d73 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/6925c51ca2854f3e83275bfcfe128d73 2023-04-06 17:56:26,273 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/6925c51ca2854f3e83275bfcfe128d73, entries=150, sequenceid=81, filesize=11.6 K 2023-04-06 17:56:26,279 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/8777265bbabb4980903bf7b64f33f0c1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/8777265bbabb4980903bf7b64f33f0c1 2023-04-06 17:56:26,297 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/8777265bbabb4980903bf7b64f33f0c1, entries=150, sequenceid=81, filesize=11.6 K 2023-04-06 17:56:26,300 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/b2f60c39442348d19b7c2ac181ef0259 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b2f60c39442348d19b7c2ac181ef0259 2023-04-06 17:56:26,318 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b2f60c39442348d19b7c2ac181ef0259, entries=150, sequenceid=81, filesize=11.6 K 2023-04-06 17:56:26,321 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 715ms, sequenceid=81, compaction requested=false 2023-04-06 17:56:26,321 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:26,321 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:26,321 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:26,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:26,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:26,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:26,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524be71b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:26,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:26,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,339 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,339 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:26,339 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:26,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:26,340 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:26,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:26,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:26,367 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:26,373 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:26,373 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:26,373 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:26,373 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:26,373 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:26,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,377 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:26,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:26,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,380 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:26,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:26,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,381 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:26,381 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:26,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:26,481 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:26,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:26,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:56:26,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:26,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:26,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:26,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:26,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:26,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:26,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:26,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:26,512 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:26,512 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:26,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:26,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:26,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,518 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:26,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:26,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:26,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803846533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803846533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803846535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803846535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:26,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:26,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:26,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:26,559 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:26,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:26,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:26,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:26,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:26,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:26,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:26,567 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:26,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:26,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:26,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:26,580 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,581 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,581 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a90b8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:26,583 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:26,583 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,584 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,584 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:26,584 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:26,584 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:26,587 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,587 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:26,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:26,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,588 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:26,590 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:26,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:26,594 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:26,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:26,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:26,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:26,594 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:26,597 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,597 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,597 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:26,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:26,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:26,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:26,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,617 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:26,618 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:26,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:26,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803846640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803846641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803846642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803846649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,718 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:26,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:26,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:26,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:26,727 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:26,727 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:26,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:26,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:26,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:26,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,733 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,733 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:26,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:26,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:26,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:26,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:26,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:26,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:26,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:26,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,737 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:26,737 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:26,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,741 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,741 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,741 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:26,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f574e5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:26,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:26,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,756 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:26,756 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:26,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:26,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:26,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:26,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:26,773 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:26,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:26,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:26,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:26,774 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:26,775 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,775 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,775 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,777 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,779 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:26,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:26,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:26,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:26,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,783 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:26,783 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:26,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803846848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803846849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803846848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803846853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,884 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:26,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:26,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:26,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:26,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:26,893 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:26,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:26,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:26,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:26,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,896 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:26,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:26,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:26,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:26,901 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:26,902 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:26,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:26,907 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:26,907 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:26,907 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:26,907 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:26,914 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,914 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:26,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e7abc95[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:26,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:26,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,924 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,924 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:26,924 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:26,924 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:26,925 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,925 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:26,926 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,926 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:26,926 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,926 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:26,936 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:26,937 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:26,937 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:26,937 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:26,937 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:26,940 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,940 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,940 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:26,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,944 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,949 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:26,949 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:26,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:26,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:26,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:26,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:26,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:26,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:26,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:26,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:26,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:26,952 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:26,952 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:26,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:26,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803846974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:26,979 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:26,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4daffd5d61f945379698baafd6898f02 2023-04-06 17:56:27,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/308537c28bb0421799d91042dac5ab57 2023-04-06 17:56:27,052 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:27,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:27,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:27,062 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:27,062 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:27,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:27,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,069 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,069 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:27,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:27,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:27,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:27,090 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:27,090 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,090 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:27,090 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:27,090 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:27,090 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,090 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:27,090 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:27,090 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,091 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:27,091 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:27,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/3d8ef0b6e4d745c28a2939ba9ba69ee6 2023-04-06 17:56:27,092 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,093 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,093 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:27,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1304e37e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:27,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:27,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,096 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,096 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,096 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:27,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,096 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:27,096 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:27,096 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:27,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:27,099 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4daffd5d61f945379698baafd6898f02 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4daffd5d61f945379698baafd6898f02 2023-04-06 17:56:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:27,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:27,105 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:27,105 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:27,105 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:27,105 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:27,105 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:27,115 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,115 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,115 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4daffd5d61f945379698baafd6898f02, entries=200, sequenceid=98, filesize=13.9 K 2023-04-06 17:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:27,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:27,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,124 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:27,124 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:27,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/308537c28bb0421799d91042dac5ab57 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/308537c28bb0421799d91042dac5ab57 2023-04-06 17:56:27,126 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/308537c28bb0421799d91042dac5ab57, entries=150, sequenceid=98, filesize=11.6 K 2023-04-06 17:56:27,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/3d8ef0b6e4d745c28a2939ba9ba69ee6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3d8ef0b6e4d745c28a2939ba9ba69ee6 2023-04-06 17:56:27,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803847153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3d8ef0b6e4d745c28a2939ba9ba69ee6, entries=150, sequenceid=98, filesize=11.6 K 2023-04-06 17:56:27,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803847154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803847155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 691ms, sequenceid=98, compaction requested=true 2023-04-06 17:56:27,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:27,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:27,176 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:27,176 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:27,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:27,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:27,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:27,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:56:27,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:27,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:27,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:27,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:27,178 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:27,178 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:27,183 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:27,183 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:27,183 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,183 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0df11ab894b4f1895696367df44136b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/6925c51ca2854f3e83275bfcfe128d73, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4daffd5d61f945379698baafd6898f02] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=37.2 K 2023-04-06 17:56:27,184 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:27,184 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:27,184 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,184 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c1206061ff9a4dd19f7d1a86450146b1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/8777265bbabb4980903bf7b64f33f0c1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/308537c28bb0421799d91042dac5ab57] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=34.9 K 2023-04-06 17:56:27,185 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b0df11ab894b4f1895696367df44136b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803783936 2023-04-06 17:56:27,194 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c1206061ff9a4dd19f7d1a86450146b1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803783936 2023-04-06 17:56:27,197 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 8777265bbabb4980903bf7b64f33f0c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680803784637 2023-04-06 17:56:27,197 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6925c51ca2854f3e83275bfcfe128d73, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680803784637 2023-04-06 17:56:27,200 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 308537c28bb0421799d91042dac5ab57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803785855 2023-04-06 17:56:27,201 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4daffd5d61f945379698baafd6898f02, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803785855 2023-04-06 17:56:27,224 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#19 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:27,225 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:27,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:27,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:27,236 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#20 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:27,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:27,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:27,253 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:27,253 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:27,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:27,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,257 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,259 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,259 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:27,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:27,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:27,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:27,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,291 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:27,291 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:27,291 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:27,292 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,292 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:27,293 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:27,293 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,294 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:27,294 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:27,294 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803847292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:27,317 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,318 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,318 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463e4ef6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:27,320 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:27,320 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,323 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:27,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:27,323 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:27,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:27,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:27,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:27,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:27,340 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:27,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:27,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:27,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:27,341 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:27,342 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,342 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,342 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,348 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:27,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:27,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,349 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:27,350 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:27,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:27,374 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/fced5d0aa5144f68b738156d12ec37ab as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fced5d0aa5144f68b738156d12ec37ab 2023-04-06 17:56:27,385 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into fced5d0aa5144f68b738156d12ec37ab(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:27,386 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:27,386 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=13, startTime=1680803787176; duration=0sec 2023-04-06 17:56:27,386 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:27,386 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:27,390 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:27,390 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:27,390 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,390 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ce36887642d4ff099451fa934d9af0e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b2f60c39442348d19b7c2ac181ef0259, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3d8ef0b6e4d745c28a2939ba9ba69ee6] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=34.9 K 2023-04-06 17:56:27,392 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 7ce36887642d4ff099451fa934d9af0e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803783936 2023-04-06 17:56:27,393 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b2f60c39442348d19b7c2ac181ef0259, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680803784637 2023-04-06 17:56:27,394 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3d8ef0b6e4d745c28a2939ba9ba69ee6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803785855 2023-04-06 17:56:27,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803847411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,415 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#21 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:27,450 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:27,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:27,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:27,458 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:27,458 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:27,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:27,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,460 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:27,465 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/d1d946f9fe7343bfb7271b6601e7f78d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1d946f9fe7343bfb7271b6601e7f78d 2023-04-06 17:56:27,475 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into d1d946f9fe7343bfb7271b6601e7f78d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:27,475 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:27,475 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=13, startTime=1680803787177; duration=0sec 2023-04-06 17:56:27,476 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:27,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:27,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:27,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:27,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:27,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:27,484 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,484 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:27,484 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:27,484 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:27,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:27,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:27,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:27,486 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:27,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f854888[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:27,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:27,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,491 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:27,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:27,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:27,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:27,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,492 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:27,492 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,492 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:27,497 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:27,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:27,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:27,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:27,497 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:27,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,516 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:27,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:27,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,522 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:27,523 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:27,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:27,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803847621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,623 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:27,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:27,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:27,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:27,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:27,627 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:27,628 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,629 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:27,629 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:27,629 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,629 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,630 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,630 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,631 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:27,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:27,632 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:27,632 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,632 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:27,633 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,633 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:27,633 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:27,633 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:27,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:27,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6450d7f9[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:27,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:27,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,652 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:27,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:27,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:27,652 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,652 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:27,653 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:27,659 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:27,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:27,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:27,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:27,659 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:27,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803847669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,671 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803847670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,675 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,676 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,676 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,676 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,676 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:27,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:27,677 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:27,677 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:27,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803847675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/2001ca9f654a4da7b5fb617c35d570bd 2023-04-06 17:56:27,753 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/c213805441044deb9239087976937a45 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c213805441044deb9239087976937a45 2023-04-06 17:56:27,773 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into c213805441044deb9239087976937a45(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:27,774 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:27,774 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=13, startTime=1680803787176; duration=0sec 2023-04-06 17:56:27,774 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:27,778 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:27,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:27,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:27,784 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:27,784 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:27,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:27,788 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,789 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,791 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,793 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,793 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:27,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:27,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:27,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:27,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,797 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:27,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/da23cd8a595145838d4f877faee92f7c 2023-04-06 17:56:27,798 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:27,798 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:27,798 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,798 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:27,798 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:27,798 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,798 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:27,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:27,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:27,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:27,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50fbc71e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:27,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:27,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,817 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,817 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:27,817 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:27,817 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:27,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:27,823 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:27,823 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,823 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:27,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:27,829 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:27,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:27,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:27,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:27,829 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:27,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/cca25bc8cd2e4868920f23bca9c7152d 2023-04-06 17:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/2001ca9f654a4da7b5fb617c35d570bd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2001ca9f654a4da7b5fb617c35d570bd 2023-04-06 17:56:27,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:27,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:27,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:27,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:27,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,857 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:27,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,859 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:27,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2001ca9f654a4da7b5fb617c35d570bd, entries=150, sequenceid=117, filesize=11.6 K 2023-04-06 17:56:27,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/da23cd8a595145838d4f877faee92f7c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/da23cd8a595145838d4f877faee92f7c 2023-04-06 17:56:27,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/da23cd8a595145838d4f877faee92f7c, entries=150, sequenceid=117, filesize=11.6 K 2023-04-06 17:56:27,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/cca25bc8cd2e4868920f23bca9c7152d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/cca25bc8cd2e4868920f23bca9c7152d 2023-04-06 17:56:27,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/cca25bc8cd2e4868920f23bca9c7152d, entries=150, sequenceid=117, filesize=11.6 K 2023-04-06 17:56:27,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 713ms, sequenceid=117, compaction requested=false 2023-04-06 17:56:27,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:27,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:27,928 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 17:56:27,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:27,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:27,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:27,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:27,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:27,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:27,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f749eb0cc27e4a19ab230027d952bbda 2023-04-06 17:56:27,959 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:27,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:27,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:27,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:27,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:27,964 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:27,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:27,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:27,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,968 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:27,968 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:27,968 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,969 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:27,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:27,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:27,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:27,972 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:27,972 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:27,973 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,973 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:27,973 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:27,973 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:27,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:27,982 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64067a09[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:27,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:27,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:27,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:27,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:27,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:27,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:27,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:27,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:27,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:27,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:27,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:27,996 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:27,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:27,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:27,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:27,996 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:27,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:27,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,000 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/7d43ffe95c8b4543b5b327e0c285a0f0 2023-04-06 17:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,011 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,011 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,011 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,012 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:28,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:28,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,017 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:28,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:28,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803848038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/3b5dd9974a4a405f99371ead0f78e066 2023-04-06 17:56:28,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f749eb0cc27e4a19ab230027d952bbda as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f749eb0cc27e4a19ab230027d952bbda 2023-04-06 17:56:28,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f749eb0cc27e4a19ab230027d952bbda, entries=200, sequenceid=134, filesize=14.0 K 2023-04-06 17:56:28,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/7d43ffe95c8b4543b5b327e0c285a0f0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7d43ffe95c8b4543b5b327e0c285a0f0 2023-04-06 17:56:28,112 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7d43ffe95c8b4543b5b327e0c285a0f0, entries=150, sequenceid=134, filesize=11.7 K 2023-04-06 17:56:28,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/3b5dd9974a4a405f99371ead0f78e066 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3b5dd9974a4a405f99371ead0f78e066 2023-04-06 17:56:28,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:28,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:28,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:28,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:28,133 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:28,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:28,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:28,136 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,136 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,141 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:28,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:28,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:28,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:28,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:28,149 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,149 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:28,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3b5dd9974a4a405f99371ead0f78e066, entries=150, sequenceid=134, filesize=11.7 K 2023-04-06 17:56:28,155 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:28,155 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:28,156 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 228ms, sequenceid=134, compaction requested=true 2023-04-06 17:56:28,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:28,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:28,156 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:28,156 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:28,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:28,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:28,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:28,158 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,160 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:28,160 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:28,160 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,160 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c213805441044deb9239087976937a45, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/da23cd8a595145838d4f877faee92f7c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7d43ffe95c8b4543b5b327e0c285a0f0] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=35.1 K 2023-04-06 17:56:28,160 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38304 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,161 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c213805441044deb9239087976937a45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803785855 2023-04-06 17:56:28,160 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:28,163 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,163 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fced5d0aa5144f68b738156d12ec37ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2001ca9f654a4da7b5fb617c35d570bd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f749eb0cc27e4a19ab230027d952bbda] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=37.4 K 2023-04-06 17:56:28,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:28,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6142af09[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:28,164 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:28,165 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,164 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting fced5d0aa5144f68b738156d12ec37ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803785855 2023-04-06 17:56:28,164 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting da23cd8a595145838d4f877faee92f7c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680803786528 2023-04-06 17:56:28,166 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 2001ca9f654a4da7b5fb617c35d570bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680803786528 2023-04-06 17:56:28,166 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7d43ffe95c8b4543b5b327e0c285a0f0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803787279 2023-04-06 17:56:28,167 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f749eb0cc27e4a19ab230027d952bbda, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803787273 2023-04-06 17:56:28,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:28,171 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:56:28,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:28,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:28,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:28,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:28,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:28,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:28,172 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:28,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:28,174 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:28,186 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:28,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:28,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:28,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:28,186 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:28,189 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:28,190 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,190 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,190 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,204 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#29 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,230 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:28,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:28,235 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:28,235 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,235 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:28,235 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:28,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,335 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:28,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:28,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:28,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:28,342 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:28,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:28,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:28,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,348 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:28,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:28,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:28,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:28,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:28,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:28,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:28,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803848357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,378 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/9853ffbef5364b0c88bd9da9c2067d62 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9853ffbef5364b0c88bd9da9c2067d62 2023-04-06 17:56:28,381 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:28,381 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:28,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:28,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:28,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:28,378 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f54cf1c6140043bfbba5f7b428705b78 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f54cf1c6140043bfbba5f7b428705b78 2023-04-06 17:56:28,389 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,395 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,395 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf53c7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:28,399 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:28,399 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:28,400 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,400 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:28,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:28,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:28,401 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:28,402 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,402 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,402 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:28,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:28,409 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:28,409 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:28,409 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:28,409 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:28,409 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:28,411 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 9853ffbef5364b0c88bd9da9c2067d62(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:28,411 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:28,411 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=13, startTime=1680803788156; duration=0sec 2023-04-06 17:56:28,411 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into f54cf1c6140043bfbba5f7b428705b78(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:28,411 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:28,411 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:28,411 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=13, startTime=1680803788156; duration=0sec 2023-04-06 17:56:28,411 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:28,411 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:28,413 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:28,413 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:28,413 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,413 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1d946f9fe7343bfb7271b6601e7f78d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/cca25bc8cd2e4868920f23bca9c7152d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3b5dd9974a4a405f99371ead0f78e066] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=35.1 K 2023-04-06 17:56:28,414 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d1d946f9fe7343bfb7271b6601e7f78d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803785855 2023-04-06 17:56:28,415 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cca25bc8cd2e4868920f23bca9c7152d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680803786528 2023-04-06 17:56:28,416 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3b5dd9974a4a405f99371ead0f78e066, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803787279 2023-04-06 17:56:28,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,419 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,432 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#30 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:28,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:28,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:28,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:28,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:28,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803848463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,490 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/c0a730775ed049008864676f189b3e5f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c0a730775ed049008864676f189b3e5f 2023-04-06 17:56:28,521 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into c0a730775ed049008864676f189b3e5f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:28,521 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:28,526 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=13, startTime=1680803788157; duration=0sec 2023-04-06 17:56:28,526 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:28,538 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:28,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:28,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:28,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:28,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:28,545 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:28,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:28,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:28,547 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:28,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:28,570 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:28,570 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,570 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:28,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:28,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:28,576 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,576 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:28,576 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:28,576 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:28,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:28,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:28,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:28,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,586 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d82a797[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:28,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:28,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,590 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,591 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,591 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:28,591 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:28,591 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:28,591 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:28,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:28,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:28,592 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:28,595 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:28,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:28,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:28,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:28,595 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:28,596 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,596 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,597 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,601 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:28,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:28,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,635 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:28,635 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:28,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:28,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803848678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803848678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803848682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803848690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,735 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:28,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:28,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:28,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:28,742 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:28,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:28,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:28,743 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,743 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,743 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,744 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:28,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:28,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:28,762 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:28,762 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,762 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:28,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:28,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:28,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:28,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:28,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,768 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/500ce7d35e5a463f946725eac03962d3 2023-04-06 17:56:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:28,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a83521c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:28,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:28,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:28,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:28,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:28,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:28,790 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:28,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,790 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,791 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:28,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:28,795 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:28,796 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:28,796 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:28,796 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:28,796 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:28,798 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,799 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,799 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,800 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,801 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:28,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:28,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:28,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:28,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,907 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:28,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:28,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:28,918 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:28,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:28,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:28,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,921 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:28,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:28,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:28,932 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:28,932 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:28,932 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:28,947 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:28,947 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:28,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:28,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:28,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:28,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:28,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:28,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:28,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:28,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,952 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ecd84a2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:28,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:28,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,955 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:28,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:28,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:28,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:28,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:28,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:28,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:28,961 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:28,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:28,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:28,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:28,961 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:28,970 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,970 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,970 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,970 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:28,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:28,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:28,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,982 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:28,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803848983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:28,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:28,988 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:28,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:28,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:28,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:28,989 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:28,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:28,989 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:28,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:28,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:28,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:28,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,089 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:29,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:29,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:29,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:29,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:29,098 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:29,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:29,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:29,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,099 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:29,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:29,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:29,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:29,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:29,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:29,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:29,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:29,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:29,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:29,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57e5bbd7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:29,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:29,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:29,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:29,126 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:29,130 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:29,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:29,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:29,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:29,130 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:29,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,132 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,135 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:29,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:29,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,135 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:29,135 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:29,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/f1def47ca25a4595b5a572ce22bd7bfb 2023-04-06 17:56:29,236 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:29,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:29,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:29,240 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:29,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:29,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:29,241 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,241 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,241 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,241 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:29,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:29,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:29,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:29,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:29,249 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:29,249 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,249 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:29,249 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:29,249 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:29,249 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:29,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:29,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:29,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:29,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc310a3[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:29,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:29,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,259 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,259 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,259 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:29,259 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:29,259 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:29,259 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:29,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:29,260 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:29,262 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:29,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:29,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:29,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:29,263 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:29,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,266 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:29,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:29,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,269 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:29,269 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:29,369 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:29,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:29,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:29,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:29,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:29,374 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:29,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:29,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:29,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:29,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:29,378 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:29,378 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,378 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:29,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:29,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:29,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:29,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:29,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:29,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:29,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:29,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:29,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f90a8f5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:29,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:29,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:29,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:29,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:29,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:29,402 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:29,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:29,405 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:29,405 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:29,405 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:29,405 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:29,405 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:29,415 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,416 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,417 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,422 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,422 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,422 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,422 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,423 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,423 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,423 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,423 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:29,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:29,425 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:29,425 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:29,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:29,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55788 deadline: 1680803849489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,525 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:29,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:29,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:29,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:29,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:29,529 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:29,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:29,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:29,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:29,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:29,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:29,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:29,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:29,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:29,534 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,534 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:29,534 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:29,534 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:29,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:29,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:29,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:29,539 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:29,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:29,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3ac531[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:29,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:29,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,552 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,553 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:29,553 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:29,553 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:29,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:29,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:29,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:29,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:29,560 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:29,560 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:29,560 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:29,560 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:29,560 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:29,573 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,573 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,574 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:29,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:29,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,583 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,583 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:29,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:29,583 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:29,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/87244bdfec704a819ac270ec75e66be5 2023-04-06 17:56:29,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/500ce7d35e5a463f946725eac03962d3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/500ce7d35e5a463f946725eac03962d3 2023-04-06 17:56:29,669 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/500ce7d35e5a463f946725eac03962d3, entries=150, sequenceid=153, filesize=11.8 K 2023-04-06 17:56:29,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/f1def47ca25a4595b5a572ce22bd7bfb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f1def47ca25a4595b5a572ce22bd7bfb 2023-04-06 17:56:29,683 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:29,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:29,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:29,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:29,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:29,687 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:29,687 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,687 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:29,687 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:29,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,689 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:29,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:29,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:29,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:29,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:29,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:29,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f1def47ca25a4595b5a572ce22bd7bfb, entries=150, sequenceid=153, filesize=11.8 K 2023-04-06 17:56:29,701 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,701 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:29,701 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:29,701 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:29,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:29,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,703 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:29,703 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:29,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/87244bdfec704a819ac270ec75e66be5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/87244bdfec704a819ac270ec75e66be5 2023-04-06 17:56:29,704 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,704 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,705 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,705 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:29,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377be442[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:29,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:29,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,708 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,708 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:29,708 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:29,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:29,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:29,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:29,710 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:29,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/87244bdfec704a819ac270ec75e66be5, entries=150, sequenceid=153, filesize=11.8 K 2023-04-06 17:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 1542ms, sequenceid=153, compaction requested=false 2023-04-06 17:56:29,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:29,716 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:29,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:29,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:29,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:29,716 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:29,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:29,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:29,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,726 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:29,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:29,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:29,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:29,731 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:29,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:29,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:29,731 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:29,731 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:29,831 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:29,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:29,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:29,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:29,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:29,835 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:29,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:29,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:29,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:29,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:29,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:29,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:29,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:29,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:29,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:29,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:29,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:29,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:29,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:29,838 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:29,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:30,247 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/a1471d6b71e9420781e710b38feef16f 2023-04-06 17:56:30,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:30,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:30,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803850548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:30,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:30,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803850654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:30,671 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/b5588346ea51486cb1f55d06c3d4beb9 2023-04-06 17:56:30,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:30,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803850683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:30,691 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:30,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:30,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803850703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:30,706 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:30,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:30,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803850708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:30,710 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803850857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:31,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803851016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,019 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8241 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:31,093 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/a36751042d324239b6f2d6791efb4e45 2023-04-06 17:56:31,101 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/a1471d6b71e9420781e710b38feef16f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/a1471d6b71e9420781e710b38feef16f 2023-04-06 17:56:31,117 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/a1471d6b71e9420781e710b38feef16f, entries=150, sequenceid=169, filesize=11.8 K 2023-04-06 17:56:31,121 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/b5588346ea51486cb1f55d06c3d4beb9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b5588346ea51486cb1f55d06c3d4beb9 2023-04-06 17:56:31,130 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b5588346ea51486cb1f55d06c3d4beb9, entries=150, sequenceid=169, filesize=11.8 K 2023-04-06 17:56:31,131 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/a36751042d324239b6f2d6791efb4e45 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a36751042d324239b6f2d6791efb4e45 2023-04-06 17:56:31,140 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a36751042d324239b6f2d6791efb4e45, entries=150, sequenceid=169, filesize=11.8 K 2023-04-06 17:56:31,141 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 1303ms, sequenceid=169, compaction requested=true 2023-04-06 17:56:31,141 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:31,141 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:31,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60eabc06[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:31,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:31,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,149 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,149 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:31,149 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:31,149 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:31,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:31,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:31,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:31,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:31,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:31,154 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:31,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:31,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:31,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:31,154 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:31,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,163 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:31,166 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:56:31,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:31,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:31,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:31,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:31,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:31,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:31,172 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,172 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:31,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:31,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:31,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:31,178 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:31,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f97c07be0a4a4a12a24760c193afa430 2023-04-06 17:56:31,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:31,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803851212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/f2c1be0eda644cd8a8d47f0574538f9f 2023-04-06 17:56:31,278 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:31,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:31,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:31,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:31,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:31,306 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:31,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:31,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,308 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:31,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:31,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:31,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:31,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:31,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:31,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:31,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:31,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:31,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:31,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:31,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:31,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:31,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:31,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803851317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:31,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3767ee[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:31,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:31,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,332 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:31,332 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:31,332 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:31,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:31,339 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,340 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:31,343 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:31,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:31,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:31,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:31,343 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:31,344 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,344 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,345 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:31,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:31,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:31,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,369 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:31,369 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:31,469 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:31,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:31,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:31,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:31,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:31,490 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:31,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:31,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:31,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,492 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,493 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,493 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:31,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:31,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:31,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:31,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:31,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:31,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:31,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:31,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:31,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:31,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:31,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:31,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803851520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@713f34d4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:31,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:31,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:31,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:31,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:31,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:31,532 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:31,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:31,535 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:31,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:31,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:31,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:31,535 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:31,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,541 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:31,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:31,549 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:31,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,549 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:31,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,650 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:31,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:31,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:31,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:31,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:31,654 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:31,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:31,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:31,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,656 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/4824b45821874418b181bc82974e0e5b 2023-04-06 17:56:31,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:31,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:31,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:31,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:31,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:31,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:31,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:31,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f97c07be0a4a4a12a24760c193afa430 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f97c07be0a4a4a12a24760c193afa430 2023-04-06 17:56:31,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:31,666 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:31,667 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:31,667 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:31,667 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,667 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:31,667 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:31,667 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,667 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:31,667 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:31,670 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,678 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,678 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:31,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@190e9745[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:31,680 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:31,680 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f97c07be0a4a4a12a24760c193afa430, entries=150, sequenceid=189, filesize=11.8 K 2023-04-06 17:56:31,682 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,682 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,682 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:31,682 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:31,683 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:31,683 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:31,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/f2c1be0eda644cd8a8d47f0574538f9f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f2c1be0eda644cd8a8d47f0574538f9f 2023-04-06 17:56:31,684 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:31,684 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,684 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:31,684 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:31,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:31,688 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:31,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:31,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:31,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:31,688 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:31,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f2c1be0eda644cd8a8d47f0574538f9f, entries=150, sequenceid=189, filesize=11.8 K 2023-04-06 17:56:31,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/4824b45821874418b181bc82974e0e5b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4824b45821874418b181bc82974e0e5b 2023-04-06 17:56:31,697 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,697 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:31,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,697 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:31,698 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,706 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:31,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:31,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:31,706 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:31,706 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:31,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:31,707 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:31,707 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:31,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:31,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:31,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4824b45821874418b181bc82974e0e5b, entries=150, sequenceid=189, filesize=11.8 K 2023-04-06 17:56:31,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 544ms, sequenceid=189, compaction requested=true 2023-04-06 17:56:31,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:31,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:31,710 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:31,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:31,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:56:31,711 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:31,713 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48402 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:31,713 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:31,713 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,713 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9853ffbef5364b0c88bd9da9c2067d62, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f1def47ca25a4595b5a572ce22bd7bfb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b5588346ea51486cb1f55d06c3d4beb9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f2c1be0eda644cd8a8d47f0574538f9f] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=47.3 K 2023-04-06 17:56:31,713 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48402 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:31,713 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:31,714 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,714 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f54cf1c6140043bfbba5f7b428705b78, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/500ce7d35e5a463f946725eac03962d3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/a1471d6b71e9420781e710b38feef16f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f97c07be0a4a4a12a24760c193afa430] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=47.3 K 2023-04-06 17:56:31,714 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9853ffbef5364b0c88bd9da9c2067d62, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803787279 2023-04-06 17:56:31,715 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f1def47ca25a4595b5a572ce22bd7bfb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680803788024 2023-04-06 17:56:31,715 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b5588346ea51486cb1f55d06c3d4beb9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803788235 2023-04-06 17:56:31,715 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f54cf1c6140043bfbba5f7b428705b78, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803787279 2023-04-06 17:56:31,716 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f2c1be0eda644cd8a8d47f0574538f9f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680803790538 2023-04-06 17:56:31,716 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 500ce7d35e5a463f946725eac03962d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680803788024 2023-04-06 17:56:31,727 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting a1471d6b71e9420781e710b38feef16f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803788235 2023-04-06 17:56:31,728 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f97c07be0a4a4a12a24760c193afa430, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680803790538 2023-04-06 17:56:31,749 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#39 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:31,783 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:31,806 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:31,807 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/f7423857d9174a519d4561fd42c3aaec as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7423857d9174a519d4561fd42c3aaec 2023-04-06 17:56:31,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:31,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:31,814 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/9fa11dbe070041a7a147328e16c58896 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/9fa11dbe070041a7a147328e16c58896 2023-04-06 17:56:31,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:31,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:31,822 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:31,822 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,822 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:31,822 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:31,823 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,824 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into f7423857d9174a519d4561fd42c3aaec(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:31,824 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:31,824 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=12, startTime=1680803791710; duration=0sec 2023-04-06 17:56:31,824 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:31,824 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:31,825 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:31,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:31,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:31,825 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:31,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:31,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:31,826 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 9fa11dbe070041a7a147328e16c58896(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:31,826 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:31,826 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=12, startTime=1680803791710; duration=0sec 2023-04-06 17:56:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:31,826 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:31,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:31,827 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48402 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:31,827 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:31,827 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,827 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c0a730775ed049008864676f189b3e5f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/87244bdfec704a819ac270ec75e66be5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a36751042d324239b6f2d6791efb4e45, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4824b45821874418b181bc82974e0e5b] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=47.3 K 2023-04-06 17:56:31,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:31,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:31,829 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:31,829 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:31,829 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:56:31,829 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:31,830 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:31,830 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:31,830 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:31,830 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:31,830 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:31,830 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c0a730775ed049008864676f189b3e5f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803787279 2023-04-06 17:56:31,831 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 87244bdfec704a819ac270ec75e66be5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680803788024 2023-04-06 17:56:31,832 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a36751042d324239b6f2d6791efb4e45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803788235 2023-04-06 17:56:31,833 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4824b45821874418b181bc82974e0e5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680803790538 2023-04-06 17:56:31,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:31,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:31,843 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/33398fc14f404f468c64d5fe9972519d 2023-04-06 17:56:31,850 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#42 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:31,888 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/6f947f9ad1954cdcb4a5675f7f684b1f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6f947f9ad1954cdcb4a5675f7f684b1f 2023-04-06 17:56:31,897 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 6f947f9ad1954cdcb4a5675f7f684b1f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:31,897 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:31,897 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=12, startTime=1680803791711; duration=0sec 2023-04-06 17:56:31,897 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:31,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:31,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803851898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:32,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803852003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:32,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803852208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,271 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/a4c31eb98588401f8084ae442eccde4d 2023-04-06 17:56:32,284 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/faa6ec97550643328f73feb16486ebde 2023-04-06 17:56:32,294 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/33398fc14f404f468c64d5fe9972519d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/33398fc14f404f468c64d5fe9972519d 2023-04-06 17:56:32,300 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/33398fc14f404f468c64d5fe9972519d, entries=150, sequenceid=203, filesize=11.8 K 2023-04-06 17:56:32,302 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/a4c31eb98588401f8084ae442eccde4d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/a4c31eb98588401f8084ae442eccde4d 2023-04-06 17:56:32,307 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/a4c31eb98588401f8084ae442eccde4d, entries=150, sequenceid=203, filesize=11.8 K 2023-04-06 17:56:32,308 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/faa6ec97550643328f73feb16486ebde as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/faa6ec97550643328f73feb16486ebde 2023-04-06 17:56:32,313 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/faa6ec97550643328f73feb16486ebde, entries=150, sequenceid=203, filesize=11.8 K 2023-04-06 17:56:32,314 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 485ms, sequenceid=203, compaction requested=false 2023-04-06 17:56:32,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:32,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:32,315 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:32,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:32,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:32,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c3112ee[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:32,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:32,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,327 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:32,327 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:32,327 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:32,327 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,327 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:32,329 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:32,331 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:32,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:32,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:32,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:32,331 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:32,343 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,343 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,343 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:32,344 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:32,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:32,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:32,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:32,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:32,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:32,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:32,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:32,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,350 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:32,350 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:32,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:32,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:32,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:32,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:32,450 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:32,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:32,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:32,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:32,462 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:32,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:32,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:32,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:32,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:32,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,463 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:32,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:32,464 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,464 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:32,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:32,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:32,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:32,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:32,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:32,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:32,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:32,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:32,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:32,467 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:56:32,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:32,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:32,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:32,468 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:32,468 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:32,468 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:32,491 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/b0a78e04120f4434a83744b29194bfde 2023-04-06 17:56:32,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:32,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:32,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803852544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:32,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803852646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:32,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803852851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:32,956 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/99677f62d05b454fa9d64eaf010129da 2023-04-06 17:56:33,009 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/101003ddb1ae4291be752b1d650deec9 2023-04-06 17:56:33,022 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/b0a78e04120f4434a83744b29194bfde as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0a78e04120f4434a83744b29194bfde 2023-04-06 17:56:33,030 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0a78e04120f4434a83744b29194bfde, entries=150, sequenceid=224, filesize=11.8 K 2023-04-06 17:56:33,033 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/99677f62d05b454fa9d64eaf010129da as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/99677f62d05b454fa9d64eaf010129da 2023-04-06 17:56:33,060 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/99677f62d05b454fa9d64eaf010129da, entries=150, sequenceid=224, filesize=11.8 K 2023-04-06 17:56:33,064 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/101003ddb1ae4291be752b1d650deec9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/101003ddb1ae4291be752b1d650deec9 2023-04-06 17:56:33,077 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/101003ddb1ae4291be752b1d650deec9, entries=150, sequenceid=224, filesize=11.8 K 2023-04-06 17:56:33,080 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 613ms, sequenceid=224, compaction requested=true 2023-04-06 17:56:33,080 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:33,080 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,082 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:33,082 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:33,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:33,090 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de80891[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:33,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:33,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,097 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:33,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:33,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:33,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:33,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:33,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:33,107 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:33,117 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:33,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:33,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:33,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:33,117 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:33,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:33,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,133 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:33,133 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:33,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:33,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:56:33,159 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:33,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:33,159 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:33,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:33,159 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:33,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:33,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803853208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,233 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:33,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:33,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:33,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:33,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:33,237 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:33,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:33,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:33,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:33,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:33,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:33,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:33,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:33,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:33,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:33,239 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,239 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:33,240 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:33,240 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:33,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:33,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:33,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:33,241 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,241 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,241 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,241 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:33,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:33,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75f91a6c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:33,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:33,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:33,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:33,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:33,247 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:33,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:33,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:33,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:33,247 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:33,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,250 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,251 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,253 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,253 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,253 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,253 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,253 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:33,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:33,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,253 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,254 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,254 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,254 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,254 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,254 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:33,254 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:33,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:33,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803853311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,354 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:33,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:33,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:33,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:33,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:33,358 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:33,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:33,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:33,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,360 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:33,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:33,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:33,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:33,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:33,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:33,362 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,363 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:33,363 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:33,363 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:33,364 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,364 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,364 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,364 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18201f29[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:33,366 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:33,366 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,367 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,367 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:33,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:33,367 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:33,367 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:33,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,368 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,369 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,369 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:33,372 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:33,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:33,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:33,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:33,372 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:33,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,380 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:33,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:33,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:33,386 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:33,386 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:33,486 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:33,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:33,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:33,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:33,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:33,494 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:33,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:33,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:33,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,496 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:33,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:33,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:33,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:33,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:33,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:33,510 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:33,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:33,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:33,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:33,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:33,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:33,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,513 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,513 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:33,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:33,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c22d8a1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:33,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:33,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,516 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:33,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:33,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:33,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:33,518 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:33,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803853515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:33,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:33,523 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:33,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:33,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:33,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:33,523 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:33,525 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,525 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,525 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,540 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:33,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:33,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:33,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,541 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:33,541 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:33,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4e5e75df77d04529b52af8553ed1d5ef 2023-04-06 17:56:33,641 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:33,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:33,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:33,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:33,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:33,645 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:33,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:33,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:33,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,648 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:33,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:33,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:33,650 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:33,650 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,650 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:33,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:33,654 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,654 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:33,654 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:33,654 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:33,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:33,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133e334[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:33,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:33,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,660 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:33,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:33,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:33,661 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,661 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:33,662 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:33,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:33,665 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:33,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:33,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:33,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:33,665 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:33,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,668 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,670 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,681 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:33,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,681 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,681 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:33,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,781 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:33,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:33,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:33,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:33,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:33,785 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:33,785 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,785 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:33,785 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:33,785 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,785 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,786 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:33,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:33,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:33,790 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:33,790 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,790 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:33,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:33,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:33,799 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,799 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:33,799 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:33,799 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:33,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:33,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:33,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:33,810 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,810 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,811 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,811 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca4b304[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:33,812 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:33,812 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,813 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,813 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:33,813 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:33,813 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:33,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:33,814 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:33,814 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,814 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:33,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:33,817 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:33,817 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:33,817 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:33,817 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:33,817 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:33,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,822 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:33,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803853823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:33,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:33,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:33,833 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:33,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,934 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:33,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:33,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:33,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:33,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:33,937 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:33,937 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,937 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:33,937 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:33,938 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,938 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,938 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,938 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,939 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,939 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:33,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:33,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:33,941 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:33,941 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:33,941 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:33,943 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:33,943 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:33,944 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,944 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:33,944 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:33,944 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:33,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:33,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:33,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:33,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:33,948 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,948 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24652aee[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:33,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:33,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:33,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:33,951 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:33,953 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:33,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:33,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:33,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:33,954 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:33,954 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,954 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,955 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:33,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:33,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:33,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:33,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:33,958 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:33,958 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,958 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:33,958 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:33,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:33,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:33,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:33,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:33,958 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:33,958 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:33,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,017 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/521132d1e7fb43c3bca1785dc2b1fada 2023-04-06 17:56:34,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/c451a0d64cdc4dfdbe9477fe80b252a5 2023-04-06 17:56:34,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4e5e75df77d04529b52af8553ed1d5ef as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e5e75df77d04529b52af8553ed1d5ef 2023-04-06 17:56:34,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e5e75df77d04529b52af8553ed1d5ef, entries=200, sequenceid=237, filesize=14.1 K 2023-04-06 17:56:34,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/521132d1e7fb43c3bca1785dc2b1fada as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/521132d1e7fb43c3bca1785dc2b1fada 2023-04-06 17:56:34,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/521132d1e7fb43c3bca1785dc2b1fada, entries=150, sequenceid=237, filesize=11.8 K 2023-04-06 17:56:34,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/c451a0d64cdc4dfdbe9477fe80b252a5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c451a0d64cdc4dfdbe9477fe80b252a5 2023-04-06 17:56:34,058 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:34,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:34,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:34,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c451a0d64cdc4dfdbe9477fe80b252a5, entries=150, sequenceid=237, filesize=11.8 K 2023-04-06 17:56:34,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:34,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:34,063 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:34,064 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,064 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:34,064 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:34,064 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,064 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,065 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:34,066 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 905ms, sequenceid=237, compaction requested=true 2023-04-06 17:56:34,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:34,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:34,066 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:34,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:34,066 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:34,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:34,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:34,068 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50978 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:34,068 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:34,068 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:34,068 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,068 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:34,068 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,068 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/9fa11dbe070041a7a147328e16c58896, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/33398fc14f404f468c64d5fe9972519d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0a78e04120f4434a83744b29194bfde, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e5e75df77d04529b52af8553ed1d5ef] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=49.8 K 2023-04-06 17:56:34,068 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7423857d9174a519d4561fd42c3aaec, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/a4c31eb98588401f8084ae442eccde4d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/99677f62d05b454fa9d64eaf010129da, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/521132d1e7fb43c3bca1785dc2b1fada] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=47.4 K 2023-04-06 17:56:34,068 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 9fa11dbe070041a7a147328e16c58896, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680803790538 2023-04-06 17:56:34,068 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f7423857d9174a519d4561fd42c3aaec, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680803790538 2023-04-06 17:56:34,069 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 33398fc14f404f468c64d5fe9972519d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680803791196 2023-04-06 17:56:34,069 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a4c31eb98588401f8084ae442eccde4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680803791196 2023-04-06 17:56:34,069 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b0a78e04120f4434a83744b29194bfde, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680803791881 2023-04-06 17:56:34,069 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 99677f62d05b454fa9d64eaf010129da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680803791881 2023-04-06 17:56:34,069 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4e5e75df77d04529b52af8553ed1d5ef, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680803792528 2023-04-06 17:56:34,070 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 521132d1e7fb43c3bca1785dc2b1fada, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680803792528 2023-04-06 17:56:34,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:34,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:34,070 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,070 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:34,070 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:56:34,070 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:34,070 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:34,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:34,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:34,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:34,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:34,092 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#52 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:34,094 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#53 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:34,118 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/b19cd3e6b7f94c668e8cb8b8116e6c0a 2023-04-06 17:56:34,138 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/46ace761760145109da45c28dc0bd151 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/46ace761760145109da45c28dc0bd151 2023-04-06 17:56:34,146 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 46ace761760145109da45c28dc0bd151(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:34,146 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:34,146 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=12, startTime=1680803794066; duration=0sec 2023-04-06 17:56:34,146 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:34,146 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:34,150 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:34,151 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:34,151 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,151 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6f947f9ad1954cdcb4a5675f7f684b1f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/faa6ec97550643328f73feb16486ebde, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/101003ddb1ae4291be752b1d650deec9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c451a0d64cdc4dfdbe9477fe80b252a5] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=47.4 K 2023-04-06 17:56:34,151 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6f947f9ad1954cdcb4a5675f7f684b1f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680803790538 2023-04-06 17:56:34,152 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting faa6ec97550643328f73feb16486ebde, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680803791196 2023-04-06 17:56:34,152 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 101003ddb1ae4291be752b1d650deec9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680803791881 2023-04-06 17:56:34,153 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c451a0d64cdc4dfdbe9477fe80b252a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680803792528 2023-04-06 17:56:34,161 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/76c0a72b5b4e40ec96ec8b3426db0ee0 2023-04-06 17:56:34,171 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:34,240 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/15533e0ea1864a078f13ac9a4885e2af as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/15533e0ea1864a078f13ac9a4885e2af 2023-04-06 17:56:34,252 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 15533e0ea1864a078f13ac9a4885e2af(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:34,252 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:34,252 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=12, startTime=1680803794067; duration=0sec 2023-04-06 17:56:34,252 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:34,259 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/d0af23185bd84b3989f1793bd12d9b10 2023-04-06 17:56:34,269 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/b19cd3e6b7f94c668e8cb8b8116e6c0a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b19cd3e6b7f94c668e8cb8b8116e6c0a 2023-04-06 17:56:34,281 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b19cd3e6b7f94c668e8cb8b8116e6c0a, entries=150, sequenceid=256, filesize=11.8 K 2023-04-06 17:56:34,283 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/76c0a72b5b4e40ec96ec8b3426db0ee0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/76c0a72b5b4e40ec96ec8b3426db0ee0 2023-04-06 17:56:34,292 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/76c0a72b5b4e40ec96ec8b3426db0ee0, entries=150, sequenceid=256, filesize=11.8 K 2023-04-06 17:56:34,293 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/d0af23185bd84b3989f1793bd12d9b10 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d0af23185bd84b3989f1793bd12d9b10 2023-04-06 17:56:34,307 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d0af23185bd84b3989f1793bd12d9b10, entries=150, sequenceid=256, filesize=11.8 K 2023-04-06 17:56:34,310 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 240ms, sequenceid=256, compaction requested=false 2023-04-06 17:56:34,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:34,311 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:34,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,312 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,313 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e280913[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:34,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:34,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,315 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:34,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:34,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:34,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:34,317 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:34,317 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,317 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:34,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:34,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:34,326 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:34,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:34,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:34,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:34,326 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:34,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,335 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,339 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,339 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,339 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,339 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:34,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:34,339 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,340 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,340 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,340 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,340 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,340 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:34,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,340 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:34,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:34,385 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:56:34,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:34,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:34,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:34,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:34,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:34,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:34,440 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:34,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:34,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:34,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:34,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:34,462 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:34,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:34,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:34,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,464 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,464 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:34,464 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:34,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:34,478 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:34,478 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,478 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:34,484 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,484 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:34,485 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:34,485 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:34,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:34,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:34,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:34,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,486 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@109c61a1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:34,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:34,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:34,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:34,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:34,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:34,491 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:34,494 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:34,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:34,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:34,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:34,494 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:34,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,498 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:34,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:34,508 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,508 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,508 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:34,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:34,508 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:34,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,555 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f6c5948841f449a3b69ab453acba3a74 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f6c5948841f449a3b69ab453acba3a74 2023-04-06 17:56:34,571 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into f6c5948841f449a3b69ab453acba3a74(size=12.3 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:34,572 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:34,572 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=12, startTime=1680803794066; duration=0sec 2023-04-06 17:56:34,572 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:34,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803794533, max timestamp=1680803794573Num uniques 859; 2023-04-06 17:56:34,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:34,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803794533, max timestamp=1680803794573Num uniques 859; 2023-04-06 17:56:34,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:34,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803794533, max timestamp=1680803794573Num uniques 859; 2023-04-06 17:56:34,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:34,608 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:34,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:34,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:34,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:34,614 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:34,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:34,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:34,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:34,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:34,630 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:34,630 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,630 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:34,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:34,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:34,639 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,639 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:34,639 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:34,640 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:34,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:34,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@939dc25[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:34,643 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:34,643 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,645 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,645 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:34,645 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:34,645 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:34,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:34,646 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:34,646 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:34,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:34,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:34,651 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:34,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:34,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:34,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:34,651 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:34,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:34,666 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:34,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:34,669 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:34,669 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:34,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:34,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803854671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:34,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55776 deadline: 1680803854709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,713 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:34,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:34,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55844 deadline: 1680803854715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,717 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:34,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:34,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55856 deadline: 1680803854727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,734 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8198 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:34,769 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:34,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:34,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:34,780 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:34,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:34,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:34,781 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,781 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:34,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:34,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:34,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:34,784 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,784 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:34,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:34,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803854782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,786 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:34,786 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,786 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:34,786 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:34,786 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,786 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:34,786 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:34,792 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486b5b6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:34,794 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:34,794 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,796 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,797 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:34,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:34,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:34,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:34,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:34,808 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:34,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:34,811 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:34,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:34,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:34,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:34,811 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:34,813 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,813 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,813 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:34,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,816 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:34,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:34,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:34,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:34,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:34,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,818 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:34,818 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:34,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:34,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:34,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/e2712dd877ca4d0b8cece53d6272a9c0 2023-04-06 17:56:34,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/e3b47a32beb74ec1ab03a06f94fb54d6 2023-04-06 17:56:34,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/a6e998aac2a545f596fc05d97acce205 2023-04-06 17:56:34,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/e2712dd877ca4d0b8cece53d6272a9c0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e2712dd877ca4d0b8cece53d6272a9c0 2023-04-06 17:56:34,918 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:34,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:34,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e2712dd877ca4d0b8cece53d6272a9c0, entries=150, sequenceid=268, filesize=11.9 K 2023-04-06 17:56:34,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:34,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/e3b47a32beb74ec1ab03a06f94fb54d6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e3b47a32beb74ec1ab03a06f94fb54d6 2023-04-06 17:56:34,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:34,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:34,930 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:34,930 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,930 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:34,930 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:34,931 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,931 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,931 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:34,932 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:34,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e3b47a32beb74ec1ab03a06f94fb54d6, entries=150, sequenceid=268, filesize=11.9 K 2023-04-06 17:56:34,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/a6e998aac2a545f596fc05d97acce205 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a6e998aac2a545f596fc05d97acce205 2023-04-06 17:56:34,934 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:34,934 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:34,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:34,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:34,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:34,938 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:34,938 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:34,938 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:34,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a6e998aac2a545f596fc05d97acce205, entries=150, sequenceid=268, filesize=11.9 K 2023-04-06 17:56:34,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 559ms, sequenceid=268, compaction requested=true 2023-04-06 17:56:34,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:34,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:34,944 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:34,944 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:34,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:34,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:34,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:34,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:34,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:34,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36838 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:34,945 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 17:56:34,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:34,945 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36838 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:34,946 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,946 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:34,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:34,946 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f6c5948841f449a3b69ab453acba3a74, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b19cd3e6b7f94c668e8cb8b8116e6c0a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e2712dd877ca4d0b8cece53d6272a9c0] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=36.0 K 2023-04-06 17:56:34,946 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:34,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:34,946 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/46ace761760145109da45c28dc0bd151, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/76c0a72b5b4e40ec96ec8b3426db0ee0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e3b47a32beb74ec1ab03a06f94fb54d6] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=36.0 K 2023-04-06 17:56:34,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:34,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:34,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:34,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:34,946 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f6c5948841f449a3b69ab453acba3a74, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680803792528 2023-04-06 17:56:34,946 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 46ace761760145109da45c28dc0bd151, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680803792528 2023-04-06 17:56:34,947 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b19cd3e6b7f94c668e8cb8b8116e6c0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680803793194 2023-04-06 17:56:34,947 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 76c0a72b5b4e40ec96ec8b3426db0ee0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680803793194 2023-04-06 17:56:34,947 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting e2712dd877ca4d0b8cece53d6272a9c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680803794352 2023-04-06 17:56:34,947 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e3b47a32beb74ec1ab03a06f94fb54d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680803794352 2023-04-06 17:56:34,967 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:34,971 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#61 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:34,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:34,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:35,034 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/e00fa1bdf0f9461189af5708c13e87ca as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e00fa1bdf0f9461189af5708c13e87ca 2023-04-06 17:56:35,035 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/e8f7acbbac7a47c7b94278431a9ca472 2023-04-06 17:56:35,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:35,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803855031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,044 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into e00fa1bdf0f9461189af5708c13e87ca(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:35,044 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,045 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=13, startTime=1680803794944; duration=0sec 2023-04-06 17:56:35,045 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:35,045 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:35,046 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36838 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:35,046 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:35,046 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,046 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/15533e0ea1864a078f13ac9a4885e2af, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d0af23185bd84b3989f1793bd12d9b10, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a6e998aac2a545f596fc05d97acce205] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=36.0 K 2023-04-06 17:56:35,047 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 15533e0ea1864a078f13ac9a4885e2af, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680803792528 2023-04-06 17:56:35,048 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d0af23185bd84b3989f1793bd12d9b10, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680803793194 2023-04-06 17:56:35,049 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a6e998aac2a545f596fc05d97acce205, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680803794352 2023-04-06 17:56:35,061 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#64 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:35,086 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/f7ad5c2ddc0546a1862d188fdf2688af 2023-04-06 17:56:35,105 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/b695fd7d4ca64863b52f762d603408ad 2023-04-06 17:56:35,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/e8f7acbbac7a47c7b94278431a9ca472 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e8f7acbbac7a47c7b94278431a9ca472 2023-04-06 17:56:35,118 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e8f7acbbac7a47c7b94278431a9ca472, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 17:56:35,119 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/f7ad5c2ddc0546a1862d188fdf2688af as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7ad5c2ddc0546a1862d188fdf2688af 2023-04-06 17:56:35,126 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7ad5c2ddc0546a1862d188fdf2688af, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 17:56:35,128 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/b695fd7d4ca64863b52f762d603408ad as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b695fd7d4ca64863b52f762d603408ad 2023-04-06 17:56:35,135 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b695fd7d4ca64863b52f762d603408ad, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 17:56:35,136 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 191ms, sequenceid=306, compaction requested=false 2023-04-06 17:56:35,137 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,137 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:35,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:35,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:35,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:35,138 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa854a4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:35,140 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:35,140 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:35,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:35,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:35,142 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,142 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:35,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:35,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:56:35,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:35,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:35,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:35,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:35,148 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:35,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:35,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:35,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:35,149 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:35,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,162 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,165 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:35,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:35,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,165 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,165 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,166 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:35,166 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:35,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4fe22d7af9de4a54ba27941e59f90b7a 2023-04-06 17:56:35,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/6dd2bda3526f4ad7b5b410fc35d6d17f 2023-04-06 17:56:35,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803795197, max timestamp=1680803795222Num uniques 859; 2023-04-06 17:56:35,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:35,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803795197, max timestamp=1680803795222Num uniques 859; 2023-04-06 17:56:35,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:35,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803795197, max timestamp=1680803795222Num uniques 859; 2023-04-06 17:56:35,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:56:35,266 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:35,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:35,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:35,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:35,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:35,303 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:35,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:35,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:35,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,304 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,304 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:35,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:35,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:35,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:35,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:35,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:35,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:35,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/5daa8bad7ea344ec863ad8bc353faf15 2023-04-06 17:56:35,316 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:35,316 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:35,316 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:35,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4fe22d7af9de4a54ba27941e59f90b7a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4fe22d7af9de4a54ba27941e59f90b7a 2023-04-06 17:56:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:35,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394ab539[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:35,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:35,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:35,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:35,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:35,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:35,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4fe22d7af9de4a54ba27941e59f90b7a, entries=200, sequenceid=318, filesize=14.3 K 2023-04-06 17:56:35,334 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/6dd2bda3526f4ad7b5b410fc35d6d17f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/6dd2bda3526f4ad7b5b410fc35d6d17f 2023-04-06 17:56:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:35,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:35,341 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:35,342 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:35,342 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:35,342 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:35,342 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:35,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/6dd2bda3526f4ad7b5b410fc35d6d17f, entries=150, sequenceid=318, filesize=11.9 K 2023-04-06 17:56:35,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/5daa8bad7ea344ec863ad8bc353faf15 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/5daa8bad7ea344ec863ad8bc353faf15 2023-04-06 17:56:35,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,357 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/5daa8bad7ea344ec863ad8bc353faf15, entries=150, sequenceid=318, filesize=11.9 K 2023-04-06 17:56:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,361 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=106.95 KB/109515 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 217ms, sequenceid=318, compaction requested=true 2023-04-06 17:56:35,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:35,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:56:35,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-06 17:56:35,364 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,364 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,364 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:35,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:35,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:35,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,364 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,365 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:35,364 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,365 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:35,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:35,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=113.66 KB heapSize=324.30 KB 2023-04-06 17:56:35,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:35,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:35,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:35,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:35,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:35,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:35,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/27026799ef494fab91a45564ef74d498 2023-04-06 17:56:35,425 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/c399557c0fc0421eae7c7c39ae551ab0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c399557c0fc0421eae7c7c39ae551ab0 2023-04-06 17:56:35,433 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into c399557c0fc0421eae7c7c39ae551ab0(size=12.6 K), total size for store is 38.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:35,433 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,433 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=13, startTime=1680803794944; duration=0sec 2023-04-06 17:56:35,433 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-06 17:56:35,433 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-06 17:56:35,435 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:56:35,435 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:56:35,435 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. because compaction request was cancelled 2023-04-06 17:56:35,435 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:35,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/202af7ded8644ca0809c4c11fc99d57b 2023-04-06 17:56:35,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:35,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803855438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,444 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:35,444 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:35,444 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,444 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c399557c0fc0421eae7c7c39ae551ab0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e8f7acbbac7a47c7b94278431a9ca472, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4fe22d7af9de4a54ba27941e59f90b7a] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=38.8 K 2023-04-06 17:56:35,445 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c399557c0fc0421eae7c7c39ae551ab0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680803794352 2023-04-06 17:56:35,445 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting e8f7acbbac7a47c7b94278431a9ca472, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803794634 2023-04-06 17:56:35,446 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4fe22d7af9de4a54ba27941e59f90b7a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680803795004 2023-04-06 17:56:35,465 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:35,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:35,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:35,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:35,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:35,475 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:35,475 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#72 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:35,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/554f87abe21b435c92c3d1e607e2a294 2023-04-06 17:56:35,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:35,476 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:35,476 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,476 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,477 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,478 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,479 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,479 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:35,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:35,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/27026799ef494fab91a45564ef74d498 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/27026799ef494fab91a45564ef74d498 2023-04-06 17:56:35,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:35,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:35,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:35,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/27026799ef494fab91a45564ef74d498, entries=200, sequenceid=352, filesize=14.3 K 2023-04-06 17:56:35,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/202af7ded8644ca0809c4c11fc99d57b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/202af7ded8644ca0809c4c11fc99d57b 2023-04-06 17:56:35,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:35,495 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:35,495 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,495 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:35,495 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:35,495 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,495 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:35,495 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:35,495 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,495 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:35,495 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:35,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:35,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e558ec1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:35,521 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:35,521 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,522 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:35,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:35,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:35,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:35,523 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/202af7ded8644ca0809c4c11fc99d57b, entries=150, sequenceid=352, filesize=11.9 K 2023-04-06 17:56:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:35,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/554f87abe21b435c92c3d1e607e2a294 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/554f87abe21b435c92c3d1e607e2a294 2023-04-06 17:56:35,526 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:35,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:35,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:35,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:35,526 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:35,527 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,527 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,527 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,532 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,533 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:35,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:35,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:35,534 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/49744696ff83471b8792ab80ca545173 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/49744696ff83471b8792ab80ca545173 2023-04-06 17:56:35,534 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:35,534 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:35,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/554f87abe21b435c92c3d1e607e2a294, entries=150, sequenceid=352, filesize=11.9 K 2023-04-06 17:56:35,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~113.66 KB/116385, heapSize ~324.26 KB/332040, currentSize=67.09 KB/68700 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 166ms, sequenceid=352, compaction requested=true 2023-04-06 17:56:35,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:56:35,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-06 17:56:35,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-06 17:56:35,544 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 49744696ff83471b8792ab80ca545173(size=12.6 K), total size for store is 48.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:35,544 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,544 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=13, startTime=1680803794944; duration=0sec 2023-04-06 17:56:35,544 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-06 17:56:35,544 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 3 compacting, 1 eligible, 16 blocking 2023-04-06 17:56:35,545 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/542e1b7f29354e16b1a79afa80066b0f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/542e1b7f29354e16b1a79afa80066b0f 2023-04-06 17:56:35,545 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:56:35,546 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:56:35,546 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. because compaction request was cancelled 2023-04-06 17:56:35,546 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:35,548 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:35,549 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:35,549 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,549 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/49744696ff83471b8792ab80ca545173, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b695fd7d4ca64863b52f762d603408ad, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/5daa8bad7ea344ec863ad8bc353faf15, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/554f87abe21b435c92c3d1e607e2a294] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=48.3 K 2023-04-06 17:56:35,549 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 49744696ff83471b8792ab80ca545173, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680803794352 2023-04-06 17:56:35,550 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b695fd7d4ca64863b52f762d603408ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803794634 2023-04-06 17:56:35,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:35,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 17:56:35,551 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5daa8bad7ea344ec863ad8bc353faf15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680803795023 2023-04-06 17:56:35,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:35,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:35,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:35,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,552 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 554f87abe21b435c92c3d1e607e2a294, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680803795326 2023-04-06 17:56:35,554 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 542e1b7f29354e16b1a79afa80066b0f(size=12.7 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:35,554 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,554 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=13, startTime=1680803795362; duration=0sec 2023-04-06 17:56:35,554 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:56:35,554 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:35,558 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:35,558 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:35,558 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,558 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e00fa1bdf0f9461189af5708c13e87ca, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7ad5c2ddc0546a1862d188fdf2688af, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/6dd2bda3526f4ad7b5b410fc35d6d17f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/202af7ded8644ca0809c4c11fc99d57b] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=48.3 K 2023-04-06 17:56:35,560 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting e00fa1bdf0f9461189af5708c13e87ca, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680803794352 2023-04-06 17:56:35,561 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f7ad5c2ddc0546a1862d188fdf2688af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803794634 2023-04-06 17:56:35,561 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6dd2bda3526f4ad7b5b410fc35d6d17f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680803795023 2023-04-06 17:56:35,562 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 202af7ded8644ca0809c4c11fc99d57b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680803795326 2023-04-06 17:56:35,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4e846465bc9e4a948f3f0bf0e557ba8d 2023-04-06 17:56:35,592 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:35,593 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#74 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:35,634 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:35,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:35,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:35,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:35,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803855654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:35,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:35,675 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:35,675 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,675 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:35,675 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:35,676 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,676 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,677 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:35,682 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/0cd23ab101b140dca358722f47d57fd2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/0cd23ab101b140dca358722f47d57fd2 2023-04-06 17:56:35,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:35,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:35,693 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:35,693 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,693 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:35,695 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 0cd23ab101b140dca358722f47d57fd2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:35,695 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,695 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=12, startTime=1680803795537; duration=0sec 2023-04-06 17:56:35,695 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:35,695 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-06 17:56:35,696 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:56:35,696 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:56:35,696 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. because compaction request was cancelled 2023-04-06 17:56:35,697 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:35,698 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:35,698 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:35,698 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:35,698 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/72c64c5e006b47869acb2e94ef076150 2023-04-06 17:56:35,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:35,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:35,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,706 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:35,706 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:35,707 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,707 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:35,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:35,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcfafb8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:35,715 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:35,715 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:35,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:35,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:35,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:35,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:35,726 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:35,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:35,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:35,729 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:35,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:35,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:35,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:35,730 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:35,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,733 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:35,736 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:35,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:35,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:35,736 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:35,736 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:35,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:35,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803855758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/1c4b15513b944745a0929aaab64eac29 2023-04-06 17:56:35,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/4e846465bc9e4a948f3f0bf0e557ba8d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e846465bc9e4a948f3f0bf0e557ba8d 2023-04-06 17:56:35,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e846465bc9e4a948f3f0bf0e557ba8d, entries=200, sequenceid=368, filesize=14.3 K 2023-04-06 17:56:35,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/72c64c5e006b47869acb2e94ef076150 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/72c64c5e006b47869acb2e94ef076150 2023-04-06 17:56:35,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/72c64c5e006b47869acb2e94ef076150, entries=150, sequenceid=368, filesize=11.9 K 2023-04-06 17:56:35,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/1c4b15513b944745a0929aaab64eac29 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/1c4b15513b944745a0929aaab64eac29 2023-04-06 17:56:35,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/1c4b15513b944745a0929aaab64eac29, entries=150, sequenceid=368, filesize=11.9 K 2023-04-06 17:56:35,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 277ms, sequenceid=368, compaction requested=true 2023-04-06 17:56:35,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:35,828 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:35,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:35,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:56:35,829 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42222 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:35,830 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:35,830 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,830 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/542e1b7f29354e16b1a79afa80066b0f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/27026799ef494fab91a45564ef74d498, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e846465bc9e4a948f3f0bf0e557ba8d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=41.2 K 2023-04-06 17:56:35,830 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 542e1b7f29354e16b1a79afa80066b0f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680803795023 2023-04-06 17:56:35,831 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 27026799ef494fab91a45564ef74d498, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680803795326 2023-04-06 17:56:35,832 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4e846465bc9e4a948f3f0bf0e557ba8d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680803795429 2023-04-06 17:56:35,836 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:35,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:35,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:35,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:35,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:35,850 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:35,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:35,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:35,852 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:35,853 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,853 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,854 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:35,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:35,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:35,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:35,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:35,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:35,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:35,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:35,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:35,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:35,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:35,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:35,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:35,875 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,882 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:35,882 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:56:35,882 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:35,882 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,882 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:35,882 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,882 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:35,882 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:35,891 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/d6dbac7717324679a9e465e02f772163 2023-04-06 17:56:35,917 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/299cf13937c1463094465127291b6a9d 2023-04-06 17:56:35,939 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/3932fd25f1824bceb1d5a42541684a4c 2023-04-06 17:56:35,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/d6dbac7717324679a9e465e02f772163 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d6dbac7717324679a9e465e02f772163 2023-04-06 17:56:35,956 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d6dbac7717324679a9e465e02f772163, entries=150, sequenceid=388, filesize=11.9 K 2023-04-06 17:56:35,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/299cf13937c1463094465127291b6a9d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/299cf13937c1463094465127291b6a9d 2023-04-06 17:56:35,964 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/299cf13937c1463094465127291b6a9d, entries=150, sequenceid=388, filesize=11.9 K 2023-04-06 17:56:35,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:35,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:35,966 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/3932fd25f1824bceb1d5a42541684a4c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3932fd25f1824bceb1d5a42541684a4c 2023-04-06 17:56:35,975 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3932fd25f1824bceb1d5a42541684a4c, entries=150, sequenceid=388, filesize=11.9 K 2023-04-06 17:56:35,976 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=13.42 KB/13740 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 94ms, sequenceid=388, compaction requested=true 2023-04-06 17:56:35,976 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:35,976 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:35,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:35,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:35,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:35,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:35,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:35,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@474acf91[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:35,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:35,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,982 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:35,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:35,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:35,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:35,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:35,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,984 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:35,984 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:35,984 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:35,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:35,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:35,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:35,995 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:35,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:35,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:35,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:35,995 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:35,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:35,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:36,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:36,005 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:36,005 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:36,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:36,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:56:36,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:36,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:36,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:36,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:36,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:36,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:36,045 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/182ce2e394f54227940dcec3b77d156f 2023-04-06 17:56:36,078 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/7578216ea6cc41888973fae3a40696b0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7578216ea6cc41888973fae3a40696b0 2023-04-06 17:56:36,085 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 7578216ea6cc41888973fae3a40696b0(size=12.7 K), total size for store is 36.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:36,085 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:36,085 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=12, startTime=1680803795537; duration=0sec 2023-04-06 17:56:36,085 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:56:36,085 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:36,087 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:36,087 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:36,087 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,087 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7578216ea6cc41888973fae3a40696b0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/1c4b15513b944745a0929aaab64eac29, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3932fd25f1824bceb1d5a42541684a4c] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=36.5 K 2023-04-06 17:56:36,088 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7578216ea6cc41888973fae3a40696b0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680803795326 2023-04-06 17:56:36,088 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 1c4b15513b944745a0929aaab64eac29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680803795431 2023-04-06 17:56:36,089 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3932fd25f1824bceb1d5a42541684a4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680803795576 2023-04-06 17:56:36,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/79c33c07c80b486bb2241b4bb8b62fb6 2023-04-06 17:56:36,103 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:36,105 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:36,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:36,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:36,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:36,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:36,122 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:36,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:36,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:36,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,126 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:36,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:36,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:36,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:36,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:36,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:36,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:36,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:36,137 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,137 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:36,137 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:36,137 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:36,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:36,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803796114, max timestamp=1680803796124Num uniques 859; 2023-04-06 17:56:36,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:36,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:36,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:36,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803796114, max timestamp=1680803796124Num uniques 859; 2023-04-06 17:56:36,144 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:36,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803796114, max timestamp=1680803796124Num uniques 859; 2023-04-06 17:56:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:36,145 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/0829e0c51ea94acebc28d2c54d83065e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/0829e0c51ea94acebc28d2c54d83065e 2023-04-06 17:56:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/d1e1c83cb86c40d6a9cd79a85b0f1919 2023-04-06 17:56:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:36,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b45593c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:36,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:36,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,155 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 0829e0c51ea94acebc28d2c54d83065e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:36,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/182ce2e394f54227940dcec3b77d156f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/182ce2e394f54227940dcec3b77d156f 2023-04-06 17:56:36,155 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:36,155 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=13, startTime=1680803795828; duration=0sec 2023-04-06 17:56:36,155 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:36,155 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:36,156 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,156 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:36,156 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:36,156 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:36,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:36,158 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:36,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:36,158 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:36,158 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:36,158 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,158 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/0cd23ab101b140dca358722f47d57fd2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/72c64c5e006b47869acb2e94ef076150, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/299cf13937c1463094465127291b6a9d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=36.5 K 2023-04-06 17:56:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,159 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0cd23ab101b140dca358722f47d57fd2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680803795326 2023-04-06 17:56:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,159 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 72c64c5e006b47869acb2e94ef076150, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680803795431 2023-04-06 17:56:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:36,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:36,161 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:36,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:36,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:36,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:36,161 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:36,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,163 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/182ce2e394f54227940dcec3b77d156f, entries=200, sequenceid=398, filesize=14.3 K 2023-04-06 17:56:36,168 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 299cf13937c1463094465127291b6a9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680803795576 2023-04-06 17:56:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,184 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/79c33c07c80b486bb2241b4bb8b62fb6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/79c33c07c80b486bb2241b4bb8b62fb6 2023-04-06 17:56:36,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,184 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:36,185 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:36,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/79c33c07c80b486bb2241b4bb8b62fb6, entries=100, sequenceid=398, filesize=9.5 K 2023-04-06 17:56:36,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/d1e1c83cb86c40d6a9cd79a85b0f1919 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1e1c83cb86c40d6a9cd79a85b0f1919 2023-04-06 17:56:36,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1e1c83cb86c40d6a9cd79a85b0f1919, entries=100, sequenceid=398, filesize=9.5 K 2023-04-06 17:56:36,203 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#86 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:36,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=80.51 KB/82440 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 192ms, sequenceid=398, compaction requested=false 2023-04-06 17:56:36,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:36,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:36,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=87.22 KB heapSize=245.53 KB 2023-04-06 17:56:36,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:36,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:36,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:36,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:36,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:36,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:36,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/22032f5b1acd4d97801728e77b847452 2023-04-06 17:56:36,283 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/63bc7e11b4a644e2907d56b9e27f1308 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/63bc7e11b4a644e2907d56b9e27f1308 2023-04-06 17:56:36,285 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:36,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:36,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:36,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:36,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:36,309 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:36,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:36,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:36,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,312 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 63bc7e11b4a644e2907d56b9e27f1308(size=12.8 K), total size for store is 22.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:36,312 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:36,312 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=13, startTime=1680803795828; duration=0sec 2023-04-06 17:56:36,312 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,312 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:36,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:36,313 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:36,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:36,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803856326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:36,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:36,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:36,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:36,361 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/d507ee554679455486847cff15c6e915 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d507ee554679455486847cff15c6e915 2023-04-06 17:56:36,368 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into d507ee554679455486847cff15c6e915(size=12.8 K), total size for store is 38.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:36,368 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:36,368 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=13, startTime=1680803795828; duration=0sec 2023-04-06 17:56:36,368 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:36,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:36,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:36,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:36,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:36,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:36,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:36,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:36,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:36,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:36,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79b4e913[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:36,387 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:36,387 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,388 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:36,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:36,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:36,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:36,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:36,389 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:36,392 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:36,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:36,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:36,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:36,393 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:36,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,403 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:36,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:36,407 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,408 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:36,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,408 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:36,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:36,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:36,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803856432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,508 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:36,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:36,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:36,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:36,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:36,514 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:36,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:36,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:36,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,515 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:36,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:36,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:36,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:36,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:36,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:36,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:36,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:36,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:36,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:36,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:36,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:36,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:36,520 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:36,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bb43bf2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:36,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:36,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,525 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:36,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:36,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:36,527 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:36,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:36,530 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:36,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:36,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:36,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:36,530 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:36,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,540 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:36,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:36,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,540 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:36,540 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,641 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:36,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:36,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:36,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803856637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:36,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:36,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:36,654 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:36,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:36,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:36,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,655 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:36,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:36,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:36,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:36,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:36,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:36,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:36,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:36,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:36,665 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,665 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:36,666 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:36,666 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:36,674 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,674 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a935352[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:36,679 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:36,679 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,682 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,682 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:36,682 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:36,682 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:36,682 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,682 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:36,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:36,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:36,685 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:36,688 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:36,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:36,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:36,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:36,688 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:36,691 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,691 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,691 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:36,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:36,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,698 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:36,698 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:36,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/9fd77c41f1774dd893e59d4730ae040d 2023-04-06 17:56:36,798 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:36,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:36,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:36,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:36,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:36,802 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:36,803 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,803 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:36,803 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:36,803 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,803 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,804 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:36,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:36,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:36,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:36,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:36,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:36,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:36,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:36,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9e7d85[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:36,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:36,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,810 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,810 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:36,810 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:36,810 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:36,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,811 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:36,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,811 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,811 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:36,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:36,814 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:36,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:36,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:36,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:36,814 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:36,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,824 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,825 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,825 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:36,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:36,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:36,833 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:36,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:36,933 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:36,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:36,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:36,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:36,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:36,940 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:36,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:36,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:36,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,941 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:36,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:36,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:36,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803856945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:36,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:36,948 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:36,948 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:36,948 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:36,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:36,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:36,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:36,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:36,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:36,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:36,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:36,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:36,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:36,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:36,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bcfb94[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:36,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:36,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:36,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:36,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:36,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:36,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:36,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:36,961 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:36,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:36,965 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:36,965 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:36,965 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:36,965 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:36,965 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:36,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:36,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,970 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:36,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:36,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:36,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:36,976 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:36,976 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:36,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:36,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:36,977 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:36,975 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:36,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:36,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:36,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:36,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:37,077 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:37,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:37,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:37,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:37,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:37,080 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:37,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:37,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:37,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:37,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,081 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:37,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:37,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:37,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:37,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:37,082 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:37,082 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:37,082 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:37,082 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:37,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:37,083 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,083 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:37,083 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:37,083 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:37,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:37,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:37,083 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:37,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:37,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:37,086 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fbd2b7a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:37,086 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:37,087 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,087 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:37,088 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:37,088 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,088 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:37,088 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:37,090 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:37,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:37,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:37,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:37,090 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:37,090 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,090 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:37,090 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:37,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:37,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:37,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:37,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:37,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,095 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:37,095 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:37,195 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:37,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:37,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:37,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:37,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:37,199 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:37,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:37,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:37,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:37,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:37,201 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:37,201 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,201 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:37,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:37,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:37,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:37,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,203 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:37,203 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:37,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/77f7883aaa3a4d72a7b81f40528a227b 2023-04-06 17:56:37,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:37,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:37,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:37,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/22032f5b1acd4d97801728e77b847452 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/22032f5b1acd4d97801728e77b847452 2023-04-06 17:56:37,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:37,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff4fc1c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:37,214 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:37,214 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,219 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:37,219 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:37,219 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:37,219 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,219 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:37,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/22032f5b1acd4d97801728e77b847452, entries=200, sequenceid=430, filesize=14.3 K 2023-04-06 17:56:37,220 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:37,220 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:37,220 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:37,220 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:37,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/9fd77c41f1774dd893e59d4730ae040d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9fd77c41f1774dd893e59d4730ae040d 2023-04-06 17:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9fd77c41f1774dd893e59d4730ae040d, entries=150, sequenceid=430, filesize=11.9 K 2023-04-06 17:56:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/77f7883aaa3a4d72a7b81f40528a227b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/77f7883aaa3a4d72a7b81f40528a227b 2023-04-06 17:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:37,231 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:37,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:37,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:37,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:37,231 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:37,233 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,233 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:37,233 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:37,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:37,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/77f7883aaa3a4d72a7b81f40528a227b, entries=150, sequenceid=430, filesize=11.9 K 2023-04-06 17:56:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:37,239 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~245.48 KB/251376, currentSize=93.93 KB/96180 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 1028ms, sequenceid=430, compaction requested=true 2023-04-06 17:56:37,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:37,239 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:37,239 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:56:37,239 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:37,239 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:37,239 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:37,241 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:37,241 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:37,241 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,241 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/63bc7e11b4a644e2907d56b9e27f1308, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/79c33c07c80b486bb2241b4bb8b62fb6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9fd77c41f1774dd893e59d4730ae040d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=34.2 K 2023-04-06 17:56:37,241 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54506 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:56:37,241 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,241 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,241 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 63bc7e11b4a644e2907d56b9e27f1308, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680803795576 2023-04-06 17:56:37,241 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d507ee554679455486847cff15c6e915, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d6dbac7717324679a9e465e02f772163, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/182ce2e394f54227940dcec3b77d156f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/22032f5b1acd4d97801728e77b847452] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=53.2 K 2023-04-06 17:56:37,242 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 79c33c07c80b486bb2241b4bb8b62fb6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680803795984 2023-04-06 17:56:37,242 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d507ee554679455486847cff15c6e915, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680803795431 2023-04-06 17:56:37,242 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9fd77c41f1774dd893e59d4730ae040d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680803796169 2023-04-06 17:56:37,242 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d6dbac7717324679a9e465e02f772163, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680803795576 2023-04-06 17:56:37,242 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 182ce2e394f54227940dcec3b77d156f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680803795984 2023-04-06 17:56:37,243 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 22032f5b1acd4d97801728e77b847452, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680803796169 2023-04-06 17:56:37,248 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,248 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:37,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:37,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:37,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:37,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:37,250 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:37,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:37,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:37,250 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:37,250 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:37,271 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:37,272 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#90 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:37,350 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:37,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:37,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:37,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:37,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:37,357 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:37,357 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:37,357 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:37,357 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:37,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:37,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:37,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:37,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:37,374 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/c6e8a6b93feb4529916d2348c2f0f966 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c6e8a6b93feb4529916d2348c2f0f966 2023-04-06 17:56:37,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:37,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:37,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:37,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:37,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:37,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:37,374 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,374 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:37,374 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 17:56:37,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:37,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:37,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:37,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:37,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:37,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:37,385 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into c6e8a6b93feb4529916d2348c2f0f966(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:37,385 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:37,385 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=12, startTime=1680803797239; duration=0sec 2023-04-06 17:56:37,385 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:37,385 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:37,386 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:37,387 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:37,387 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:37,388 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/0829e0c51ea94acebc28d2c54d83065e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1e1c83cb86c40d6a9cd79a85b0f1919, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/77f7883aaa3a4d72a7b81f40528a227b] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=34.2 K 2023-04-06 17:56:37,389 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 0829e0c51ea94acebc28d2c54d83065e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680803795576 2023-04-06 17:56:37,389 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d1e1c83cb86c40d6a9cd79a85b0f1919, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680803795984 2023-04-06 17:56:37,390 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 77f7883aaa3a4d72a7b81f40528a227b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680803796169 2023-04-06 17:56:37,407 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:37,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:37,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:37,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:37,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803857476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:37,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803857579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,745 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/aae8fa6b93cf4da0a713ccb1b269baa8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/aae8fa6b93cf4da0a713ccb1b269baa8 2023-04-06 17:56:37,750 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into aae8fa6b93cf4da0a713ccb1b269baa8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:37,751 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:37,751 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=13, startTime=1680803797239; duration=0sec 2023-04-06 17:56:37,751 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:37,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:37,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803857782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:37,813 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/944c4eb10e844606a697d085f0cbb8be 2023-04-06 17:56:37,824 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/76bfd36835bb4962a07ec7439380f8f1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/76bfd36835bb4962a07ec7439380f8f1 2023-04-06 17:56:37,832 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 76bfd36835bb4962a07ec7439380f8f1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:37,832 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:37,832 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=13, startTime=1680803797239; duration=0sec 2023-04-06 17:56:37,832 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:38,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:38,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803858086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,238 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/b39bcb58681b44a3ba1a4ecda9ee90c0 2023-04-06 17:56:38,250 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/ca8af0b0e30b413fa7a5010c485cdab1 2023-04-06 17:56:38,255 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/944c4eb10e844606a697d085f0cbb8be as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/944c4eb10e844606a697d085f0cbb8be 2023-04-06 17:56:38,261 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/944c4eb10e844606a697d085f0cbb8be, entries=150, sequenceid=449, filesize=11.9 K 2023-04-06 17:56:38,262 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/b39bcb58681b44a3ba1a4ecda9ee90c0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b39bcb58681b44a3ba1a4ecda9ee90c0 2023-04-06 17:56:38,266 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b39bcb58681b44a3ba1a4ecda9ee90c0, entries=150, sequenceid=449, filesize=11.9 K 2023-04-06 17:56:38,267 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/ca8af0b0e30b413fa7a5010c485cdab1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ca8af0b0e30b413fa7a5010c485cdab1 2023-04-06 17:56:38,273 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ca8af0b0e30b413fa7a5010c485cdab1, entries=150, sequenceid=449, filesize=11.9 K 2023-04-06 17:56:38,275 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 900ms, sequenceid=449, compaction requested=false 2023-04-06 17:56:38,275 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:38,275 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:38,275 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:38,275 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:38,275 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,275 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:38,275 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:38,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,276 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:38,276 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,276 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ece32[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:38,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:38,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:38,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:38,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:38,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:38,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:38,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:38,281 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:38,282 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:38,282 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:38,282 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:38,282 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:38,282 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,282 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:38,283 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:38,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:38,283 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:38,284 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,287 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,287 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:38,287 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:38,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:38,287 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:38,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:38,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:38,287 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:38,287 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:38,287 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,288 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,288 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:38,288 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:38,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:38,288 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,288 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:38,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:38,388 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:38,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:38,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:38,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:38,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:38,392 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:38,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:38,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:38,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:38,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:38,393 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:38,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:38,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:38,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:38,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:38,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:38,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:38,394 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:38,394 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:38,394 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 17:56:38,395 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:38,395 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:38,395 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:38,395 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:38,395 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:38,395 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:38,404 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/b7bf606eca024abfa12bd9ec663993a7 2023-04-06 17:56:38,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:38,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:38,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:38,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803858620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:38,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803858722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:38,852 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/bdaa90b4e9ca4af1b649b580df34566e 2023-04-06 17:56:38,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:38,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803858926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:39,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803859230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,263 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/59b93d3633064e08b7d68e90d9a719cb 2023-04-06 17:56:39,268 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/b7bf606eca024abfa12bd9ec663993a7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b7bf606eca024abfa12bd9ec663993a7 2023-04-06 17:56:39,272 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b7bf606eca024abfa12bd9ec663993a7, entries=150, sequenceid=467, filesize=11.9 K 2023-04-06 17:56:39,273 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/bdaa90b4e9ca4af1b649b580df34566e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/bdaa90b4e9ca4af1b649b580df34566e 2023-04-06 17:56:39,277 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/bdaa90b4e9ca4af1b649b580df34566e, entries=150, sequenceid=467, filesize=11.9 K 2023-04-06 17:56:39,278 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/59b93d3633064e08b7d68e90d9a719cb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/59b93d3633064e08b7d68e90d9a719cb 2023-04-06 17:56:39,281 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/59b93d3633064e08b7d68e90d9a719cb, entries=150, sequenceid=467, filesize=11.9 K 2023-04-06 17:56:39,282 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 888ms, sequenceid=467, compaction requested=true 2023-04-06 17:56:39,282 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:39,282 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:39,283 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,283 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:39,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:39,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:39,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652cf518[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:39,285 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:39,285 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,285 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:39,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:39,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:39,285 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,285 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:39,286 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:39,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:39,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:39,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:39,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:39,288 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:39,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:39,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:39,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:39,288 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:39,289 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,289 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:39,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,289 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:39,289 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:39,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:39,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:39,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:39,292 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:39,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:39,292 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:39,293 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:39,393 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:39,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:39,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:39,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:39,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:39,395 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:39,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:39,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:39,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:39,397 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,397 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:39,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:39,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:39,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:39,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:39,399 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,399 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:39,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:39,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:39,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:39,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:39,399 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:39,399 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:39,399 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:39,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:39,400 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:39,400 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:39,400 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:39,412 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/061f1b6636494c1b8ad8013bab4a7b7e 2023-04-06 17:56:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:39,736 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:39,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:39,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803859764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:39,832 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/66217193cd184e7eba873bea2dd1b4e9 2023-04-06 17:56:39,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:39,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803859867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:40,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803860070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,243 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/4bd19458afc44f669105dd26cec0c3d2 2023-04-06 17:56:40,247 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/061f1b6636494c1b8ad8013bab4a7b7e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/061f1b6636494c1b8ad8013bab4a7b7e 2023-04-06 17:56:40,252 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/061f1b6636494c1b8ad8013bab4a7b7e, entries=150, sequenceid=484, filesize=11.9 K 2023-04-06 17:56:40,253 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/66217193cd184e7eba873bea2dd1b4e9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/66217193cd184e7eba873bea2dd1b4e9 2023-04-06 17:56:40,258 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/66217193cd184e7eba873bea2dd1b4e9, entries=150, sequenceid=484, filesize=11.9 K 2023-04-06 17:56:40,262 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/4bd19458afc44f669105dd26cec0c3d2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4bd19458afc44f669105dd26cec0c3d2 2023-04-06 17:56:40,267 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4bd19458afc44f669105dd26cec0c3d2, entries=150, sequenceid=484, filesize=11.9 K 2023-04-06 17:56:40,268 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 868ms, sequenceid=484, compaction requested=true 2023-04-06 17:56:40,268 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:40,268 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:40,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:40,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:40,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:40,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,270 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,270 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:40,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63cce43c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:40,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:40,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:40,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:40,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:40,272 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,272 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:40,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,273 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:40,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:40,275 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:40,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:40,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:40,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:40,275 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,276 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,277 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:40,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:40,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:40,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,279 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:40,279 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:40,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:40,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 17:56:40,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:40,375 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:40,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:40,375 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:40,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:40,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:40,379 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:40,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:40,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:40,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:40,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:40,383 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:40,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:40,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:40,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,384 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,384 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,384 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:40,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/70017088a71e475cbf3206df7923bb11 2023-04-06 17:56:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:40,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:40,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:40,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:40,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:40,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:40,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:40,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:40,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:40,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:40,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:40,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:40,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:40,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,388 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:40,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bd0e045[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:40,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:40,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,394 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,394 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:40,395 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:40,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:40,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:40,398 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:40,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:40,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:40,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:40,398 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:40,399 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,399 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,404 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:40,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:40,405 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:40,405 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:40,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:40,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803860424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,505 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:40,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:40,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:40,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:40,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:40,507 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:40,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:40,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:40,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:40,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:40,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:40,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:40,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:40,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:40,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:40,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:40,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:40,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:40,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:40,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:40,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:40,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:40,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:40,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:40,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1afeec53[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:40,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,514 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,514 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:40,514 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:40,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:40,514 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:40,515 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:40,515 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,515 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,516 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:40,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:40,518 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:40,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:40,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:40,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:40,518 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:40,519 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,519 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,519 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,519 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,520 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,521 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:40,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,522 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,522 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,522 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,522 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:40,522 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,522 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:40,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:40,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803860527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,622 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:40,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:40,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:40,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:40,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:40,625 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:40,625 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,625 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:40,625 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:40,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,627 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,627 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,627 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,627 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:40,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:40,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:40,628 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,628 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:40,628 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:40,628 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:40,628 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:40,629 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,629 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,630 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,630 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:40,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18785e21[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:40,631 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:40,631 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,632 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:40,632 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,633 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:40,633 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:40,635 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:40,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:40,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:40,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:40,635 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:40,637 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,637 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,637 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,639 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,639 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,639 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:40,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:40,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:40,640 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:40,640 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:40,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:40,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803860729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,740 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:40,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:40,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:40,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:40,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:40,744 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:40,744 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,744 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:40,744 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:40,744 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,744 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,745 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:40,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:40,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:40,746 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:40,746 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,746 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:40,746 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:40,746 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:40,746 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,746 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:40,746 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:40,746 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,746 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:40,747 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:40,747 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,747 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:40,747 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:40,747 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,747 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:40,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c91b416[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:40,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:40,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:40,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:40,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:40,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:40,750 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:40,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:40,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:40,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:40,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:40,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:40,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:40,753 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,753 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,753 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,755 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,756 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:40,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,756 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:40,756 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:40,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/198a240f17fd40a79b538e68438408ab 2023-04-06 17:56:40,857 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:40,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:40,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:40,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:40,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:40,859 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:40,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:40,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:40,860 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,860 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,861 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,861 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,862 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,862 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:40,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:40,863 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,863 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:40,863 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:40,863 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:40,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:40,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,865 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,865 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70afad09[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:40,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:40,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:40,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:40,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:40,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:40,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:40,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:40,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:40,869 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:40,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:40,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:40,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:40,869 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:40,870 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,870 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,870 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,871 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,872 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:40,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:40,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:40,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,873 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:40,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,873 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:40,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,973 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:40,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:40,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:40,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:40,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:40,976 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:40,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:40,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:40,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,977 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,977 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,977 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:40,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:40,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:40,978 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,978 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:40,978 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:40,978 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:40,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:40,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:40,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dde5b3b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:40,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:40,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,981 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:40,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:40,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:40,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:40,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:40,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,982 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:40,984 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:40,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:40,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:40,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:40,984 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:40,985 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,985 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,985 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:40,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:40,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:40,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:40,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:40,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:40,988 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:40,988 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:40,988 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:40,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:40,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:41,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803861033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:41,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55802 deadline: 1680803861039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,042 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18263 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:56:41,088 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:41,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:41,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:41,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:41,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:41,091 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:41,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:41,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:41,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,092 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,093 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,093 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:41,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:41,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:41,094 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,094 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:41,094 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:41,094 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:41,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:41,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:41,096 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:41,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@478d87f7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:41,096 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,097 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,097 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:41,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:41,097 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:41,097 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:41,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:41,098 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:41,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:41,099 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:41,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:41,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:41,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:41,099 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:41,100 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,100 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,100 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,101 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,102 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:41,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,104 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:41,104 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:41,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:41,204 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:41,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:41,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:41,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:41,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:41,207 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:41,207 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,207 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:41,207 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:41,207 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,207 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/eb5e6c11e3074f45ae8b4484c4b7d2bc 2023-04-06 17:56:41,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,224 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:41,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:41,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:41,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:41,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/70017088a71e475cbf3206df7923bb11 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/70017088a71e475cbf3206df7923bb11 2023-04-06 17:56:41,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:41,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:41,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:41,230 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:41,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:41,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@562daa14[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:41,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:41,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:41,233 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:41,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:41,235 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/70017088a71e475cbf3206df7923bb11, entries=200, sequenceid=500, filesize=14.3 K 2023-04-06 17:56:41,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:41,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:41,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/198a240f17fd40a79b538e68438408ab as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/198a240f17fd40a79b538e68438408ab 2023-04-06 17:56:41,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:41,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:41,238 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:41,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:41,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:41,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:41,239 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:41,240 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,240 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,240 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,241 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/198a240f17fd40a79b538e68438408ab, entries=150, sequenceid=500, filesize=11.9 K 2023-04-06 17:56:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/eb5e6c11e3074f45ae8b4484c4b7d2bc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/eb5e6c11e3074f45ae8b4484c4b7d2bc 2023-04-06 17:56:41,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:41,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,244 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:41,244 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:41,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:41,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:41,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/eb5e6c11e3074f45ae8b4484c4b7d2bc, entries=150, sequenceid=500, filesize=11.9 K 2023-04-06 17:56:41,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 873ms, sequenceid=500, compaction requested=true 2023-04-06 17:56:41,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:41,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:41,249 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:56:41,249 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:56:41,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:41,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:41,250 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61952 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:56:41,250 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:41,250 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,250 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64392 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:56:41,250 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/aae8fa6b93cf4da0a713ccb1b269baa8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b39bcb58681b44a3ba1a4ecda9ee90c0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/bdaa90b4e9ca4af1b649b580df34566e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/66217193cd184e7eba873bea2dd1b4e9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/198a240f17fd40a79b538e68438408ab] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=60.5 K 2023-04-06 17:56:41,250 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:41,250 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,251 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c6e8a6b93feb4529916d2348c2f0f966, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/944c4eb10e844606a697d085f0cbb8be, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b7bf606eca024abfa12bd9ec663993a7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/061f1b6636494c1b8ad8013bab4a7b7e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/70017088a71e475cbf3206df7923bb11] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=62.9 K 2023-04-06 17:56:41,251 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c6e8a6b93feb4529916d2348c2f0f966, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680803796169 2023-04-06 17:56:41,251 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting aae8fa6b93cf4da0a713ccb1b269baa8, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680803796169 2023-04-06 17:56:41,251 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 944c4eb10e844606a697d085f0cbb8be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680803796284 2023-04-06 17:56:41,251 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b39bcb58681b44a3ba1a4ecda9ee90c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680803796284 2023-04-06 17:56:41,252 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b7bf606eca024abfa12bd9ec663993a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680803797465 2023-04-06 17:56:41,252 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting bdaa90b4e9ca4af1b649b580df34566e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680803797465 2023-04-06 17:56:41,252 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 061f1b6636494c1b8ad8013bab4a7b7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680803798613 2023-04-06 17:56:41,252 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 66217193cd184e7eba873bea2dd1b4e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680803798613 2023-04-06 17:56:41,252 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 70017088a71e475cbf3206df7923bb11, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680803799754 2023-04-06 17:56:41,253 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 198a240f17fd40a79b538e68438408ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680803799754 2023-04-06 17:56:41,261 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#105 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:41,263 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#106 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:41,290 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/d79a0ba2af2a4c25891d5062a1d63fe7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d79a0ba2af2a4c25891d5062a1d63fe7 2023-04-06 17:56:41,295 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/e1359a3c6a194346ab25465d5d43ea70 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e1359a3c6a194346ab25465d5d43ea70 2023-04-06 17:56:41,299 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into d79a0ba2af2a4c25891d5062a1d63fe7(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:41,300 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:41,300 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=11, startTime=1680803801249; duration=0sec 2023-04-06 17:56:41,300 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:41,300 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:56:41,302 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61952 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:56:41,302 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:41,302 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,302 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/76bfd36835bb4962a07ec7439380f8f1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ca8af0b0e30b413fa7a5010c485cdab1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/59b93d3633064e08b7d68e90d9a719cb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4bd19458afc44f669105dd26cec0c3d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/eb5e6c11e3074f45ae8b4484c4b7d2bc] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=60.5 K 2023-04-06 17:56:41,302 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 76bfd36835bb4962a07ec7439380f8f1, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680803796169 2023-04-06 17:56:41,303 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into e1359a3c6a194346ab25465d5d43ea70(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:41,303 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:41,303 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=11, startTime=1680803801249; duration=0sec 2023-04-06 17:56:41,303 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:41,303 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ca8af0b0e30b413fa7a5010c485cdab1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680803796284 2023-04-06 17:56:41,303 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 59b93d3633064e08b7d68e90d9a719cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680803797465 2023-04-06 17:56:41,304 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4bd19458afc44f669105dd26cec0c3d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680803798613 2023-04-06 17:56:41,304 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting eb5e6c11e3074f45ae8b4484c4b7d2bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680803799754 2023-04-06 17:56:41,318 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#107 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:41,333 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/3ea1231bb38a425f9ff43a7e91983165 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3ea1231bb38a425f9ff43a7e91983165 2023-04-06 17:56:41,338 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 3ea1231bb38a425f9ff43a7e91983165(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:41,338 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:41,339 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=11, startTime=1680803801249; duration=0sec 2023-04-06 17:56:41,339 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:41,345 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:41,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:41,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:41,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:41,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:41,350 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:41,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:41,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:41,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:41,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:41,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:41,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:41,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:41,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:41,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:41,354 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,354 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:41,355 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 17:56:41,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:41,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:41,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:41,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:41,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:41,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:41,373 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f75d045c8ebe4f6983e6d26ec8fd7636 2023-04-06 17:56:41,392 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/71a471faec784dfebfee35229620574a 2023-04-06 17:56:41,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:41,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:41,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:41,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803861572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:41,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803861677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,812 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/ecfe3ec20d544d2fa17f4278abaca91f 2023-04-06 17:56:41,816 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/f75d045c8ebe4f6983e6d26ec8fd7636 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f75d045c8ebe4f6983e6d26ec8fd7636 2023-04-06 17:56:41,821 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f75d045c8ebe4f6983e6d26ec8fd7636, entries=150, sequenceid=519, filesize=11.9 K 2023-04-06 17:56:41,822 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/71a471faec784dfebfee35229620574a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/71a471faec784dfebfee35229620574a 2023-04-06 17:56:41,826 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/71a471faec784dfebfee35229620574a, entries=150, sequenceid=519, filesize=11.9 K 2023-04-06 17:56:41,827 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/ecfe3ec20d544d2fa17f4278abaca91f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ecfe3ec20d544d2fa17f4278abaca91f 2023-04-06 17:56:41,831 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ecfe3ec20d544d2fa17f4278abaca91f, entries=150, sequenceid=519, filesize=11.9 K 2023-04-06 17:56:41,832 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 477ms, sequenceid=519, compaction requested=false 2023-04-06 17:56:41,832 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:41,832 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:41,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:41,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:41,833 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:41,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,833 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,834 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,834 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:41,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b67823d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:41,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:41,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:41,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:41,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:41,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:41,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:41,839 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:41,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:41,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:41,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:41,839 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:41,840 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,840 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,840 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:41,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:41,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:41,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,844 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:41,844 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:41,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:41,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:41,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 17:56:41,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:41,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:41,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:41,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:41,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:41,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:41,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:41,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 499 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803861911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,944 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:41,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:41,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:41,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:41,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:41,947 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:41,947 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,947 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:41,947 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:41,947 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,947 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,948 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:41,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:41,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:41,949 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,949 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:41,949 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:41,949 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:41,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:41,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:41,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@348f1de9[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:41,952 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:41,952 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,952 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:41,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:41,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:41,952 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,952 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:41,953 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:41,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:41,955 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:41,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:41,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:41,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:41,955 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:41,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:41,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:41,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,957 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:41,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:41,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:41,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:41,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:41,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:41,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:41,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:41,959 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:41,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:41,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:41,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:41,959 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:41,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:41,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:41,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:42,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 501 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803862013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,060 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:42,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:42,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:42,062 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:42,062 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,062 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:42,062 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:42,063 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,063 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,063 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,063 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,064 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,064 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:42,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:42,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:42,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:42,065 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,065 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:42,065 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:42,065 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:42,065 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:42,066 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,066 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,066 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,066 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:42,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f9696b0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:42,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:42,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,068 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:42,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:42,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:42,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:42,069 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:42,071 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:42,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:42,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:42,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:42,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:42,071 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:42,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:42,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:42,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:42,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,077 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:42,077 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:42,191 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:42,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:42,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:42,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:42,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:42,194 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:42,194 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,194 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:42,194 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:42,195 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,195 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,195 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,195 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:42,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:42,197 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,197 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:42,197 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:42,197 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:42,198 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24d0f25c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:42,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:42,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,200 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:42,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:42,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:42,203 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,203 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:42,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:42,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:42,205 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:42,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:42,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:42,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:42,206 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:42,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,207 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,208 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:42,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:42,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:42,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,209 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,209 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,210 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:42,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,210 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:42,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:42,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 503 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803862215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/222c04063d964faf93ceedbd4185bd3d 2023-04-06 17:56:42,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/df9a77ab63674eb5bd25c9fbc87170c6 2023-04-06 17:56:42,310 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:42,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:42,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:42,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:42,313 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:42,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:42,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:42,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,314 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:42,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:42,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:42,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:42,315 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,315 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:42,315 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:42,315 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:42,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:42,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:42,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4341fa91[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:42,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,318 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,318 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:42,318 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:42,318 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:42,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:42,319 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:42,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:42,320 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:42,321 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:42,321 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:42,321 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:42,321 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:42,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,323 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:42,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:42,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,327 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:42,327 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:42,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,427 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:42,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:42,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:42,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:42,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:42,429 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:42,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:42,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:42,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:42,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:42,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:42,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:42,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:42,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:42,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:42,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:42,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:42,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. started... 2023-04-06 17:56:42,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. as already flushing 2023-04-06 17:56:42,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:42,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:42,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:42,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:42,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:42,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6dc745[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:42,442 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:42,442 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,442 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,443 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:42,443 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:42,443 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:42,443 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,443 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:42,443 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,443 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:42,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,444 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,444 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,444 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e18c51d to 127.0.0.1:49384 2023-04-06 17:56:42,445 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:42,446 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31222f6c to 127.0.0.1:49384 2023-04-06 17:56:42,446 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:42,447 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57966649 to 127.0.0.1:49384 2023-04-06 17:56:42,447 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:42,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:42,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:42,451 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:42,451 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:42,451 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:42,451 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:42,451 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:42,451 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09048085 to 127.0.0.1:49384 2023-04-06 17:56:42,452 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:42,452 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:42,452 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:42,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,452 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:42,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,454 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,454 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,454 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:42,454 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:42,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:42,454 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:42,454 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:42,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:42,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:42,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:42,454 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,455 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,455 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:42,455 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:42,455 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:42,455 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:42,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:42,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 505 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55788 deadline: 1680803862519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:42,555 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:42,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:42,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/aa09196256aa4b3ea12b9f82a705c3ce 2023-04-06 17:56:42,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/222c04063d964faf93ceedbd4185bd3d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/222c04063d964faf93ceedbd4185bd3d 2023-04-06 17:56:42,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/222c04063d964faf93ceedbd4185bd3d, entries=200, sequenceid=536, filesize=14.3 K 2023-04-06 17:56:42,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/df9a77ab63674eb5bd25c9fbc87170c6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/df9a77ab63674eb5bd25c9fbc87170c6 2023-04-06 17:56:42,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/df9a77ab63674eb5bd25c9fbc87170c6, entries=150, sequenceid=536, filesize=11.9 K 2023-04-06 17:56:42,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/aa09196256aa4b3ea12b9f82a705c3ce as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/aa09196256aa4b3ea12b9f82a705c3ce 2023-04-06 17:56:42,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/aa09196256aa4b3ea12b9f82a705c3ce, entries=150, sequenceid=536, filesize=11.9 K 2023-04-06 17:56:42,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 861ms, sequenceid=536, compaction requested=true 2023-04-06 17:56:42,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:42,743 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:42,743 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:42,743 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:42,743 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:42,743 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:42,744 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:42,744 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:42,744 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/A is initiating minor compaction (all files) 2023-04-06 17:56:42,744 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/B is initiating minor compaction (all files) 2023-04-06 17:56:42,744 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/A in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,744 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/B in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:42,744 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e1359a3c6a194346ab25465d5d43ea70, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/71a471faec784dfebfee35229620574a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/df9a77ab63674eb5bd25c9fbc87170c6] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=36.9 K 2023-04-06 17:56:42,744 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d79a0ba2af2a4c25891d5062a1d63fe7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f75d045c8ebe4f6983e6d26ec8fd7636, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/222c04063d964faf93ceedbd4185bd3d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=39.2 K 2023-04-06 17:56:42,745 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e1359a3c6a194346ab25465d5d43ea70, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680803799754 2023-04-06 17:56:42,745 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d79a0ba2af2a4c25891d5062a1d63fe7, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680803799754 2023-04-06 17:56:42,745 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 71a471faec784dfebfee35229620574a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680803800416 2023-04-06 17:56:42,745 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f75d045c8ebe4f6983e6d26ec8fd7636, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680803800416 2023-04-06 17:56:42,745 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting df9a77ab63674eb5bd25c9fbc87170c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680803801559 2023-04-06 17:56:42,745 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 222c04063d964faf93ceedbd4185bd3d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680803801559 2023-04-06 17:56:42,753 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#B#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:42,753 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#A#compaction#115 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:43,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:43,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 17:56:43,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:43,026 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6194be61 to 127.0.0.1:49384 2023-04-06 17:56:43,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:43,027 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:43,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:43,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:43,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:43,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:43,170 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/207a5d711ad043fc9e328ada0824ce3b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/207a5d711ad043fc9e328ada0824ce3b 2023-04-06 17:56:43,170 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/fc7dbc1a84714d898cd8608eb00f87e2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/fc7dbc1a84714d898cd8608eb00f87e2 2023-04-06 17:56:43,177 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/A of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into 207a5d711ad043fc9e328ada0824ce3b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:43,177 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/B of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into fc7dbc1a84714d898cd8608eb00f87e2(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:43,177 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:43,177 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:43,177 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/A, priority=13, startTime=1680803802743; duration=0sec 2023-04-06 17:56:43,177 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/B, priority=13, startTime=1680803802743; duration=0sec 2023-04-06 17:56:43,177 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:56:43,177 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:43,177 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:56:43,178 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:56:43,178 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 1a4cbf1c6f3aeda9a5b63ebd9e930798/C is initiating minor compaction (all files) 2023-04-06 17:56:43,178 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a4cbf1c6f3aeda9a5b63ebd9e930798/C in TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:43,178 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3ea1231bb38a425f9ff43a7e91983165, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ecfe3ec20d544d2fa17f4278abaca91f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/aa09196256aa4b3ea12b9f82a705c3ce] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp, totalSize=36.9 K 2023-04-06 17:56:43,179 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3ea1231bb38a425f9ff43a7e91983165, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680803799754 2023-04-06 17:56:43,179 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ecfe3ec20d544d2fa17f4278abaca91f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680803800416 2023-04-06 17:56:43,180 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting aa09196256aa4b3ea12b9f82a705c3ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680803801559 2023-04-06 17:56:43,190 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a4cbf1c6f3aeda9a5b63ebd9e930798#C#compaction#117 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:56:43,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/fd62bb3e28de47e391755db92e836d0f 2023-04-06 17:56:43,611 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/d21292aa5cfe431aaaedd18e93b08730 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d21292aa5cfe431aaaedd18e93b08730 2023-04-06 17:56:43,619 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a4cbf1c6f3aeda9a5b63ebd9e930798/C of 1a4cbf1c6f3aeda9a5b63ebd9e930798 into d21292aa5cfe431aaaedd18e93b08730(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:56:43,619 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:43,619 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798., storeName=1a4cbf1c6f3aeda9a5b63ebd9e930798/C, priority=13, startTime=1680803802743; duration=0sec 2023-04-06 17:56:43,619 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:56:43,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/cb7b710681374fc2a4c0aadab050dfd6 2023-04-06 17:56:44,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/84fd809dde5e4041aafe13c86ca86316 2023-04-06 17:56:44,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/fd62bb3e28de47e391755db92e836d0f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fd62bb3e28de47e391755db92e836d0f 2023-04-06 17:56:44,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fd62bb3e28de47e391755db92e836d0f, entries=150, sequenceid=552, filesize=11.9 K 2023-04-06 17:56:44,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/cb7b710681374fc2a4c0aadab050dfd6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/cb7b710681374fc2a4c0aadab050dfd6 2023-04-06 17:56:44,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/cb7b710681374fc2a4c0aadab050dfd6, entries=150, sequenceid=552, filesize=11.9 K 2023-04-06 17:56:44,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/84fd809dde5e4041aafe13c86ca86316 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/84fd809dde5e4041aafe13c86ca86316 2023-04-06 17:56:44,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/84fd809dde5e4041aafe13c86ca86316, entries=150, sequenceid=552, filesize=11.9 K 2023-04-06 17:56:44,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 1289ms, sequenceid=552, compaction requested=false 2023-04-06 17:56:44,315 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:44,741 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37b22714 to 127.0.0.1:49384 2023-04-06 17:56:44,741 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:44,764 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3339866b to 127.0.0.1:49384 2023-04-06 17:56:44,764 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:44,783 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x754370ad to 127.0.0.1:49384 2023-04-06 17:56:44,783 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:51,075 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78c6cbc8 to 127.0.0.1:49384 2023-04-06 17:56:51,076 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 361 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 17:56:51,076 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 4530 2023-04-06 17:56:51,077 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 4416 2023-04-06 17:56:51,077 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 17:56:51,077 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 1992 2023-04-06 17:56:51,077 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 5973 rows 2023-04-06 17:56:51,077 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2018 2023-04-06 17:56:51,077 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 6051 rows 2023-04-06 17:56:51,077 INFO [Listener at localhost.localdomain/40169] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 17:56:51,077 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41908139 to 127.0.0.1:49384 2023-04-06 17:56:51,077 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:56:51,082 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 17:56:51,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 17:56:51,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:51,095 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803811095"}]},"ts":"1680803811095"} 2023-04-06 17:56:51,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 17:56:51,097 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 17:56:51,098 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 17:56:51,099 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a4cbf1c6f3aeda9a5b63ebd9e930798, UNASSIGN}] 2023-04-06 17:56:51,101 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a4cbf1c6f3aeda9a5b63ebd9e930798, UNASSIGN 2023-04-06 17:56:51,102 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1a4cbf1c6f3aeda9a5b63ebd9e930798, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:51,102 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803811102"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803811102"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803811102"}]},"ts":"1680803811102"} 2023-04-06 17:56:51,103 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:56:51,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 17:56:51,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:51,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1a4cbf1c6f3aeda9a5b63ebd9e930798, disabling compactions & flushes 2023-04-06 17:56:51,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:51,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:51,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. after waiting 0 ms 2023-04-06 17:56:51,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:51,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1a4cbf1c6f3aeda9a5b63ebd9e930798 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 17:56:51,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=A 2023-04-06 17:56:51,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:51,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=B 2023-04-06 17:56:51,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:51,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a4cbf1c6f3aeda9a5b63ebd9e930798, store=C 2023-04-06 17:56:51,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:51,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 17:56:51,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/3fa2fb40864743ea9eb5a0789d1d013f 2023-04-06 17:56:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 17:56:52,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/eba2d592e82844548c296c51bf2609ec 2023-04-06 17:56:52,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/7ec8fa45c5334004a518b99ea366adcc 2023-04-06 17:56:52,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/A/3fa2fb40864743ea9eb5a0789d1d013f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/3fa2fb40864743ea9eb5a0789d1d013f 2023-04-06 17:56:52,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/3fa2fb40864743ea9eb5a0789d1d013f, entries=100, sequenceid=562, filesize=9.5 K 2023-04-06 17:56:52,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/B/eba2d592e82844548c296c51bf2609ec as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/eba2d592e82844548c296c51bf2609ec 2023-04-06 17:56:52,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/eba2d592e82844548c296c51bf2609ec, entries=100, sequenceid=562, filesize=9.5 K 2023-04-06 17:56:52,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/.tmp/C/7ec8fa45c5334004a518b99ea366adcc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ec8fa45c5334004a518b99ea366adcc 2023-04-06 17:56:52,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ec8fa45c5334004a518b99ea366adcc, entries=100, sequenceid=562, filesize=9.5 K 2023-04-06 17:56:52,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1a4cbf1c6f3aeda9a5b63ebd9e930798 in 864ms, sequenceid=562, compaction requested=true 2023-04-06 17:56:52,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/bad127d70e6a425cb33713c6edb19eac, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2170e9306a004cf4beb8e9e577fff771, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c98794e6936e4e5792ece7bd3c6f9654, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0df11ab894b4f1895696367df44136b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/6925c51ca2854f3e83275bfcfe128d73, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4daffd5d61f945379698baafd6898f02, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fced5d0aa5144f68b738156d12ec37ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2001ca9f654a4da7b5fb617c35d570bd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f749eb0cc27e4a19ab230027d952bbda, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f54cf1c6140043bfbba5f7b428705b78, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/500ce7d35e5a463f946725eac03962d3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/a1471d6b71e9420781e710b38feef16f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/9fa11dbe070041a7a147328e16c58896, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f97c07be0a4a4a12a24760c193afa430, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/33398fc14f404f468c64d5fe9972519d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0a78e04120f4434a83744b29194bfde, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e5e75df77d04529b52af8553ed1d5ef, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f6c5948841f449a3b69ab453acba3a74, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b19cd3e6b7f94c668e8cb8b8116e6c0a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c399557c0fc0421eae7c7c39ae551ab0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e2712dd877ca4d0b8cece53d6272a9c0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e8f7acbbac7a47c7b94278431a9ca472, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4fe22d7af9de4a54ba27941e59f90b7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/542e1b7f29354e16b1a79afa80066b0f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/27026799ef494fab91a45564ef74d498, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e846465bc9e4a948f3f0bf0e557ba8d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d507ee554679455486847cff15c6e915, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d6dbac7717324679a9e465e02f772163, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/182ce2e394f54227940dcec3b77d156f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/22032f5b1acd4d97801728e77b847452, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c6e8a6b93feb4529916d2348c2f0f966, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/944c4eb10e844606a697d085f0cbb8be, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b7bf606eca024abfa12bd9ec663993a7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/061f1b6636494c1b8ad8013bab4a7b7e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/70017088a71e475cbf3206df7923bb11, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d79a0ba2af2a4c25891d5062a1d63fe7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f75d045c8ebe4f6983e6d26ec8fd7636, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/222c04063d964faf93ceedbd4185bd3d] to archive 2023-04-06 17:56:52,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:56:52,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/bad127d70e6a425cb33713c6edb19eac to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/bad127d70e6a425cb33713c6edb19eac 2023-04-06 17:56:52,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2170e9306a004cf4beb8e9e577fff771 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2170e9306a004cf4beb8e9e577fff771 2023-04-06 17:56:52,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c98794e6936e4e5792ece7bd3c6f9654 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c98794e6936e4e5792ece7bd3c6f9654 2023-04-06 17:56:52,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0df11ab894b4f1895696367df44136b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0df11ab894b4f1895696367df44136b 2023-04-06 17:56:52,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/6925c51ca2854f3e83275bfcfe128d73 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/6925c51ca2854f3e83275bfcfe128d73 2023-04-06 17:56:52,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4daffd5d61f945379698baafd6898f02 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4daffd5d61f945379698baafd6898f02 2023-04-06 17:56:52,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fced5d0aa5144f68b738156d12ec37ab to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fced5d0aa5144f68b738156d12ec37ab 2023-04-06 17:56:52,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2001ca9f654a4da7b5fb617c35d570bd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/2001ca9f654a4da7b5fb617c35d570bd 2023-04-06 17:56:52,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f749eb0cc27e4a19ab230027d952bbda to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f749eb0cc27e4a19ab230027d952bbda 2023-04-06 17:56:52,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f54cf1c6140043bfbba5f7b428705b78 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f54cf1c6140043bfbba5f7b428705b78 2023-04-06 17:56:52,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/500ce7d35e5a463f946725eac03962d3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/500ce7d35e5a463f946725eac03962d3 2023-04-06 17:56:52,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/a1471d6b71e9420781e710b38feef16f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/a1471d6b71e9420781e710b38feef16f 2023-04-06 17:56:52,169 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/9fa11dbe070041a7a147328e16c58896 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/9fa11dbe070041a7a147328e16c58896 2023-04-06 17:56:52,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f97c07be0a4a4a12a24760c193afa430 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f97c07be0a4a4a12a24760c193afa430 2023-04-06 17:56:52,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/33398fc14f404f468c64d5fe9972519d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/33398fc14f404f468c64d5fe9972519d 2023-04-06 17:56:52,172 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0a78e04120f4434a83744b29194bfde to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b0a78e04120f4434a83744b29194bfde 2023-04-06 17:56:52,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e5e75df77d04529b52af8553ed1d5ef to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e5e75df77d04529b52af8553ed1d5ef 2023-04-06 17:56:52,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f6c5948841f449a3b69ab453acba3a74 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f6c5948841f449a3b69ab453acba3a74 2023-04-06 17:56:52,176 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b19cd3e6b7f94c668e8cb8b8116e6c0a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b19cd3e6b7f94c668e8cb8b8116e6c0a 2023-04-06 17:56:52,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c399557c0fc0421eae7c7c39ae551ab0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c399557c0fc0421eae7c7c39ae551ab0 2023-04-06 17:56:52,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e2712dd877ca4d0b8cece53d6272a9c0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e2712dd877ca4d0b8cece53d6272a9c0 2023-04-06 17:56:52,179 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e8f7acbbac7a47c7b94278431a9ca472 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/e8f7acbbac7a47c7b94278431a9ca472 2023-04-06 17:56:52,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4fe22d7af9de4a54ba27941e59f90b7a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4fe22d7af9de4a54ba27941e59f90b7a 2023-04-06 17:56:52,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/542e1b7f29354e16b1a79afa80066b0f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/542e1b7f29354e16b1a79afa80066b0f 2023-04-06 17:56:52,182 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/27026799ef494fab91a45564ef74d498 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/27026799ef494fab91a45564ef74d498 2023-04-06 17:56:52,184 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e846465bc9e4a948f3f0bf0e557ba8d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/4e846465bc9e4a948f3f0bf0e557ba8d 2023-04-06 17:56:52,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d507ee554679455486847cff15c6e915 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d507ee554679455486847cff15c6e915 2023-04-06 17:56:52,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d6dbac7717324679a9e465e02f772163 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d6dbac7717324679a9e465e02f772163 2023-04-06 17:56:52,187 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/182ce2e394f54227940dcec3b77d156f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/182ce2e394f54227940dcec3b77d156f 2023-04-06 17:56:52,188 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/22032f5b1acd4d97801728e77b847452 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/22032f5b1acd4d97801728e77b847452 2023-04-06 17:56:52,189 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c6e8a6b93feb4529916d2348c2f0f966 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/c6e8a6b93feb4529916d2348c2f0f966 2023-04-06 17:56:52,190 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/944c4eb10e844606a697d085f0cbb8be to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/944c4eb10e844606a697d085f0cbb8be 2023-04-06 17:56:52,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b7bf606eca024abfa12bd9ec663993a7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/b7bf606eca024abfa12bd9ec663993a7 2023-04-06 17:56:52,192 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/061f1b6636494c1b8ad8013bab4a7b7e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/061f1b6636494c1b8ad8013bab4a7b7e 2023-04-06 17:56:52,193 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/70017088a71e475cbf3206df7923bb11 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/70017088a71e475cbf3206df7923bb11 2023-04-06 17:56:52,195 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d79a0ba2af2a4c25891d5062a1d63fe7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/d79a0ba2af2a4c25891d5062a1d63fe7 2023-04-06 17:56:52,196 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f75d045c8ebe4f6983e6d26ec8fd7636 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/f75d045c8ebe4f6983e6d26ec8fd7636 2023-04-06 17:56:52,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/222c04063d964faf93ceedbd4185bd3d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/222c04063d964faf93ceedbd4185bd3d 2023-04-06 17:56:52,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 17:56:52,222 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7683ad0ead04480aa7eb1c4a67f2e810, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/315ed765b92448e4accc1ad36328b669, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c1206061ff9a4dd19f7d1a86450146b1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/300d65636d1c462ea172d0ed9a8c2bfa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/8777265bbabb4980903bf7b64f33f0c1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c213805441044deb9239087976937a45, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/308537c28bb0421799d91042dac5ab57, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/da23cd8a595145838d4f877faee92f7c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9853ffbef5364b0c88bd9da9c2067d62, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7d43ffe95c8b4543b5b327e0c285a0f0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f1def47ca25a4595b5a572ce22bd7bfb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b5588346ea51486cb1f55d06c3d4beb9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7423857d9174a519d4561fd42c3aaec, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f2c1be0eda644cd8a8d47f0574538f9f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/a4c31eb98588401f8084ae442eccde4d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/99677f62d05b454fa9d64eaf010129da, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/46ace761760145109da45c28dc0bd151, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/521132d1e7fb43c3bca1785dc2b1fada, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/76c0a72b5b4e40ec96ec8b3426db0ee0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e00fa1bdf0f9461189af5708c13e87ca, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e3b47a32beb74ec1ab03a06f94fb54d6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7ad5c2ddc0546a1862d188fdf2688af, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/6dd2bda3526f4ad7b5b410fc35d6d17f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/0cd23ab101b140dca358722f47d57fd2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/202af7ded8644ca0809c4c11fc99d57b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/72c64c5e006b47869acb2e94ef076150, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/63bc7e11b4a644e2907d56b9e27f1308, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/299cf13937c1463094465127291b6a9d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/79c33c07c80b486bb2241b4bb8b62fb6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/aae8fa6b93cf4da0a713ccb1b269baa8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9fd77c41f1774dd893e59d4730ae040d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b39bcb58681b44a3ba1a4ecda9ee90c0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/bdaa90b4e9ca4af1b649b580df34566e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/66217193cd184e7eba873bea2dd1b4e9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e1359a3c6a194346ab25465d5d43ea70, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/198a240f17fd40a79b538e68438408ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/71a471faec784dfebfee35229620574a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/df9a77ab63674eb5bd25c9fbc87170c6] to archive 2023-04-06 17:56:52,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:56:52,224 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7683ad0ead04480aa7eb1c4a67f2e810 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7683ad0ead04480aa7eb1c4a67f2e810 2023-04-06 17:56:52,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/315ed765b92448e4accc1ad36328b669 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/315ed765b92448e4accc1ad36328b669 2023-04-06 17:56:52,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c1206061ff9a4dd19f7d1a86450146b1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c1206061ff9a4dd19f7d1a86450146b1 2023-04-06 17:56:52,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/300d65636d1c462ea172d0ed9a8c2bfa to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/300d65636d1c462ea172d0ed9a8c2bfa 2023-04-06 17:56:52,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/8777265bbabb4980903bf7b64f33f0c1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/8777265bbabb4980903bf7b64f33f0c1 2023-04-06 17:56:52,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c213805441044deb9239087976937a45 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/c213805441044deb9239087976937a45 2023-04-06 17:56:52,231 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/308537c28bb0421799d91042dac5ab57 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/308537c28bb0421799d91042dac5ab57 2023-04-06 17:56:52,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/da23cd8a595145838d4f877faee92f7c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/da23cd8a595145838d4f877faee92f7c 2023-04-06 17:56:52,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9853ffbef5364b0c88bd9da9c2067d62 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9853ffbef5364b0c88bd9da9c2067d62 2023-04-06 17:56:52,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7d43ffe95c8b4543b5b327e0c285a0f0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/7d43ffe95c8b4543b5b327e0c285a0f0 2023-04-06 17:56:52,235 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f1def47ca25a4595b5a572ce22bd7bfb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f1def47ca25a4595b5a572ce22bd7bfb 2023-04-06 17:56:52,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b5588346ea51486cb1f55d06c3d4beb9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b5588346ea51486cb1f55d06c3d4beb9 2023-04-06 17:56:52,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7423857d9174a519d4561fd42c3aaec to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7423857d9174a519d4561fd42c3aaec 2023-04-06 17:56:52,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f2c1be0eda644cd8a8d47f0574538f9f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f2c1be0eda644cd8a8d47f0574538f9f 2023-04-06 17:56:52,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/a4c31eb98588401f8084ae442eccde4d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/a4c31eb98588401f8084ae442eccde4d 2023-04-06 17:56:52,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/99677f62d05b454fa9d64eaf010129da to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/99677f62d05b454fa9d64eaf010129da 2023-04-06 17:56:52,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/46ace761760145109da45c28dc0bd151 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/46ace761760145109da45c28dc0bd151 2023-04-06 17:56:52,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/521132d1e7fb43c3bca1785dc2b1fada to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/521132d1e7fb43c3bca1785dc2b1fada 2023-04-06 17:56:52,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/76c0a72b5b4e40ec96ec8b3426db0ee0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/76c0a72b5b4e40ec96ec8b3426db0ee0 2023-04-06 17:56:52,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e00fa1bdf0f9461189af5708c13e87ca to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e00fa1bdf0f9461189af5708c13e87ca 2023-04-06 17:56:52,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e3b47a32beb74ec1ab03a06f94fb54d6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e3b47a32beb74ec1ab03a06f94fb54d6 2023-04-06 17:56:52,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7ad5c2ddc0546a1862d188fdf2688af to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/f7ad5c2ddc0546a1862d188fdf2688af 2023-04-06 17:56:52,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/6dd2bda3526f4ad7b5b410fc35d6d17f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/6dd2bda3526f4ad7b5b410fc35d6d17f 2023-04-06 17:56:52,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/0cd23ab101b140dca358722f47d57fd2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/0cd23ab101b140dca358722f47d57fd2 2023-04-06 17:56:52,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/202af7ded8644ca0809c4c11fc99d57b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/202af7ded8644ca0809c4c11fc99d57b 2023-04-06 17:56:52,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/72c64c5e006b47869acb2e94ef076150 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/72c64c5e006b47869acb2e94ef076150 2023-04-06 17:56:52,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/63bc7e11b4a644e2907d56b9e27f1308 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/63bc7e11b4a644e2907d56b9e27f1308 2023-04-06 17:56:52,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/299cf13937c1463094465127291b6a9d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/299cf13937c1463094465127291b6a9d 2023-04-06 17:56:52,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/79c33c07c80b486bb2241b4bb8b62fb6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/79c33c07c80b486bb2241b4bb8b62fb6 2023-04-06 17:56:52,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/aae8fa6b93cf4da0a713ccb1b269baa8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/aae8fa6b93cf4da0a713ccb1b269baa8 2023-04-06 17:56:52,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9fd77c41f1774dd893e59d4730ae040d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/9fd77c41f1774dd893e59d4730ae040d 2023-04-06 17:56:52,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b39bcb58681b44a3ba1a4ecda9ee90c0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/b39bcb58681b44a3ba1a4ecda9ee90c0 2023-04-06 17:56:52,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/bdaa90b4e9ca4af1b649b580df34566e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/bdaa90b4e9ca4af1b649b580df34566e 2023-04-06 17:56:52,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/66217193cd184e7eba873bea2dd1b4e9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/66217193cd184e7eba873bea2dd1b4e9 2023-04-06 17:56:52,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e1359a3c6a194346ab25465d5d43ea70 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/e1359a3c6a194346ab25465d5d43ea70 2023-04-06 17:56:52,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/198a240f17fd40a79b538e68438408ab to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/198a240f17fd40a79b538e68438408ab 2023-04-06 17:56:52,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/71a471faec784dfebfee35229620574a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/71a471faec784dfebfee35229620574a 2023-04-06 17:56:52,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/df9a77ab63674eb5bd25c9fbc87170c6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/df9a77ab63674eb5bd25c9fbc87170c6 2023-04-06 17:56:52,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/50c61f132c364f698542b837fb803130, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6cb67ab7566e4d789be2d3b717c5a719, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ce36887642d4ff099451fa934d9af0e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/565a37fcfc7745c8a4002ed083c004ec, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b2f60c39442348d19b7c2ac181ef0259, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1d946f9fe7343bfb7271b6601e7f78d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3d8ef0b6e4d745c28a2939ba9ba69ee6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/cca25bc8cd2e4868920f23bca9c7152d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c0a730775ed049008864676f189b3e5f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3b5dd9974a4a405f99371ead0f78e066, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/87244bdfec704a819ac270ec75e66be5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a36751042d324239b6f2d6791efb4e45, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6f947f9ad1954cdcb4a5675f7f684b1f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4824b45821874418b181bc82974e0e5b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/faa6ec97550643328f73feb16486ebde, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/101003ddb1ae4291be752b1d650deec9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/15533e0ea1864a078f13ac9a4885e2af, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c451a0d64cdc4dfdbe9477fe80b252a5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d0af23185bd84b3989f1793bd12d9b10, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/49744696ff83471b8792ab80ca545173, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a6e998aac2a545f596fc05d97acce205, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b695fd7d4ca64863b52f762d603408ad, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/5daa8bad7ea344ec863ad8bc353faf15, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7578216ea6cc41888973fae3a40696b0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/554f87abe21b435c92c3d1e607e2a294, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/1c4b15513b944745a0929aaab64eac29, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/0829e0c51ea94acebc28d2c54d83065e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3932fd25f1824bceb1d5a42541684a4c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1e1c83cb86c40d6a9cd79a85b0f1919, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/76bfd36835bb4962a07ec7439380f8f1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/77f7883aaa3a4d72a7b81f40528a227b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ca8af0b0e30b413fa7a5010c485cdab1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/59b93d3633064e08b7d68e90d9a719cb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4bd19458afc44f669105dd26cec0c3d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3ea1231bb38a425f9ff43a7e91983165, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/eb5e6c11e3074f45ae8b4484c4b7d2bc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ecfe3ec20d544d2fa17f4278abaca91f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/aa09196256aa4b3ea12b9f82a705c3ce] to archive 2023-04-06 17:56:52,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:56:52,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/50c61f132c364f698542b837fb803130 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/50c61f132c364f698542b837fb803130 2023-04-06 17:56:52,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6cb67ab7566e4d789be2d3b717c5a719 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6cb67ab7566e4d789be2d3b717c5a719 2023-04-06 17:56:52,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ce36887642d4ff099451fa934d9af0e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ce36887642d4ff099451fa934d9af0e 2023-04-06 17:56:52,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/565a37fcfc7745c8a4002ed083c004ec to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/565a37fcfc7745c8a4002ed083c004ec 2023-04-06 17:56:52,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b2f60c39442348d19b7c2ac181ef0259 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b2f60c39442348d19b7c2ac181ef0259 2023-04-06 17:56:52,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1d946f9fe7343bfb7271b6601e7f78d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1d946f9fe7343bfb7271b6601e7f78d 2023-04-06 17:56:52,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3d8ef0b6e4d745c28a2939ba9ba69ee6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3d8ef0b6e4d745c28a2939ba9ba69ee6 2023-04-06 17:56:52,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/cca25bc8cd2e4868920f23bca9c7152d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/cca25bc8cd2e4868920f23bca9c7152d 2023-04-06 17:56:52,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c0a730775ed049008864676f189b3e5f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c0a730775ed049008864676f189b3e5f 2023-04-06 17:56:52,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3b5dd9974a4a405f99371ead0f78e066 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3b5dd9974a4a405f99371ead0f78e066 2023-04-06 17:56:52,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/87244bdfec704a819ac270ec75e66be5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/87244bdfec704a819ac270ec75e66be5 2023-04-06 17:56:52,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a36751042d324239b6f2d6791efb4e45 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a36751042d324239b6f2d6791efb4e45 2023-04-06 17:56:52,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6f947f9ad1954cdcb4a5675f7f684b1f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/6f947f9ad1954cdcb4a5675f7f684b1f 2023-04-06 17:56:52,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4824b45821874418b181bc82974e0e5b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4824b45821874418b181bc82974e0e5b 2023-04-06 17:56:52,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/faa6ec97550643328f73feb16486ebde to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/faa6ec97550643328f73feb16486ebde 2023-04-06 17:56:52,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/101003ddb1ae4291be752b1d650deec9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/101003ddb1ae4291be752b1d650deec9 2023-04-06 17:56:52,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/15533e0ea1864a078f13ac9a4885e2af to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/15533e0ea1864a078f13ac9a4885e2af 2023-04-06 17:56:52,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c451a0d64cdc4dfdbe9477fe80b252a5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/c451a0d64cdc4dfdbe9477fe80b252a5 2023-04-06 17:56:52,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d0af23185bd84b3989f1793bd12d9b10 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d0af23185bd84b3989f1793bd12d9b10 2023-04-06 17:56:52,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/49744696ff83471b8792ab80ca545173 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/49744696ff83471b8792ab80ca545173 2023-04-06 17:56:52,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a6e998aac2a545f596fc05d97acce205 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/a6e998aac2a545f596fc05d97acce205 2023-04-06 17:56:52,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b695fd7d4ca64863b52f762d603408ad to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/b695fd7d4ca64863b52f762d603408ad 2023-04-06 17:56:52,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/5daa8bad7ea344ec863ad8bc353faf15 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/5daa8bad7ea344ec863ad8bc353faf15 2023-04-06 17:56:52,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7578216ea6cc41888973fae3a40696b0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7578216ea6cc41888973fae3a40696b0 2023-04-06 17:56:52,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/554f87abe21b435c92c3d1e607e2a294 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/554f87abe21b435c92c3d1e607e2a294 2023-04-06 17:56:52,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/1c4b15513b944745a0929aaab64eac29 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/1c4b15513b944745a0929aaab64eac29 2023-04-06 17:56:52,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/0829e0c51ea94acebc28d2c54d83065e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/0829e0c51ea94acebc28d2c54d83065e 2023-04-06 17:56:52,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3932fd25f1824bceb1d5a42541684a4c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3932fd25f1824bceb1d5a42541684a4c 2023-04-06 17:56:52,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1e1c83cb86c40d6a9cd79a85b0f1919 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d1e1c83cb86c40d6a9cd79a85b0f1919 2023-04-06 17:56:52,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/76bfd36835bb4962a07ec7439380f8f1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/76bfd36835bb4962a07ec7439380f8f1 2023-04-06 17:56:52,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/77f7883aaa3a4d72a7b81f40528a227b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/77f7883aaa3a4d72a7b81f40528a227b 2023-04-06 17:56:52,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ca8af0b0e30b413fa7a5010c485cdab1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ca8af0b0e30b413fa7a5010c485cdab1 2023-04-06 17:56:52,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/59b93d3633064e08b7d68e90d9a719cb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/59b93d3633064e08b7d68e90d9a719cb 2023-04-06 17:56:52,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4bd19458afc44f669105dd26cec0c3d2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/4bd19458afc44f669105dd26cec0c3d2 2023-04-06 17:56:52,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3ea1231bb38a425f9ff43a7e91983165 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/3ea1231bb38a425f9ff43a7e91983165 2023-04-06 17:56:52,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/eb5e6c11e3074f45ae8b4484c4b7d2bc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/eb5e6c11e3074f45ae8b4484c4b7d2bc 2023-04-06 17:56:52,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ecfe3ec20d544d2fa17f4278abaca91f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/ecfe3ec20d544d2fa17f4278abaca91f 2023-04-06 17:56:52,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/aa09196256aa4b3ea12b9f82a705c3ce to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/aa09196256aa4b3ea12b9f82a705c3ce 2023-04-06 17:56:52,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/recovered.edits/565.seqid, newMaxSeqId=565, maxSeqId=1 2023-04-06 17:56:52,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798. 2023-04-06 17:56:52,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1a4cbf1c6f3aeda9a5b63ebd9e930798: 2023-04-06 17:56:52,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:52,358 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1a4cbf1c6f3aeda9a5b63ebd9e930798, regionState=CLOSED 2023-04-06 17:56:52,359 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803812358"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803812358"}]},"ts":"1680803812358"} 2023-04-06 17:56:52,363 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-06 17:56:52,363 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 1a4cbf1c6f3aeda9a5b63ebd9e930798, server=jenkins-hbase20.apache.org,38773,1680803776439 in 1.2570 sec 2023-04-06 17:56:52,366 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-06 17:56:52,366 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a4cbf1c6f3aeda9a5b63ebd9e930798, UNASSIGN in 1.2640 sec 2023-04-06 17:56:52,367 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803812367"}]},"ts":"1680803812367"} 2023-04-06 17:56:52,368 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 17:56:52,369 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 17:56:52,371 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2820 sec 2023-04-06 17:56:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 17:56:53,203 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-06 17:56:53,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 17:56:53,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:53,221 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:53,223 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-06 17:56:53,227 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:53,230 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/recovered.edits] 2023-04-06 17:56:53,233 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/207a5d711ad043fc9e328ada0824ce3b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/207a5d711ad043fc9e328ada0824ce3b 2023-04-06 17:56:53,235 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/3fa2fb40864743ea9eb5a0789d1d013f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/3fa2fb40864743ea9eb5a0789d1d013f 2023-04-06 17:56:53,236 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fd62bb3e28de47e391755db92e836d0f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/A/fd62bb3e28de47e391755db92e836d0f 2023-04-06 17:56:53,239 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/cb7b710681374fc2a4c0aadab050dfd6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/cb7b710681374fc2a4c0aadab050dfd6 2023-04-06 17:56:53,241 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/eba2d592e82844548c296c51bf2609ec to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/eba2d592e82844548c296c51bf2609ec 2023-04-06 17:56:53,242 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/fc7dbc1a84714d898cd8608eb00f87e2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/B/fc7dbc1a84714d898cd8608eb00f87e2 2023-04-06 17:56:53,244 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ec8fa45c5334004a518b99ea366adcc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/7ec8fa45c5334004a518b99ea366adcc 2023-04-06 17:56:53,246 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/84fd809dde5e4041aafe13c86ca86316 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/84fd809dde5e4041aafe13c86ca86316 2023-04-06 17:56:53,247 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d21292aa5cfe431aaaedd18e93b08730 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/C/d21292aa5cfe431aaaedd18e93b08730 2023-04-06 17:56:53,250 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/recovered.edits/565.seqid to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798/recovered.edits/565.seqid 2023-04-06 17:56:53,251 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/1a4cbf1c6f3aeda9a5b63ebd9e930798 2023-04-06 17:56:53,251 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:56:53,253 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:53,269 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 17:56:53,304 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 17:56:53,306 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:53,306 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 17:56:53,306 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680803813306"}]},"ts":"9223372036854775807"} 2023-04-06 17:56:53,308 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 17:56:53,308 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1a4cbf1c6f3aeda9a5b63ebd9e930798, NAME => 'TestAcidGuarantees,,1680803781177.1a4cbf1c6f3aeda9a5b63ebd9e930798.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 17:56:53,308 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 17:56:53,309 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680803813309"}]},"ts":"9223372036854775807"} 2023-04-06 17:56:53,311 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 17:56:53,313 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:53,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 100 msec 2023-04-06 17:56:53,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-06 17:56:53,326 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-06 17:56:53,340 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=229 (was 205) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1313749174_15 at /127.0.0.1:55872 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ef4de32-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:38773-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ef4de32-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ef4de32-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ef4de32-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=542 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 181) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 168), AvailableMemoryMB=3139 (was 3582) 2023-04-06 17:56:53,350 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229, OpenFileDescriptor=542, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=166, AvailableMemoryMB=3139 2023-04-06 17:56:53,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:56:53,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:56:53,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:53,355 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 17:56:53,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-06 17:56:53,356 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 17:56:53,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 17:56:53,358 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:53,359 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 empty. 2023-04-06 17:56:53,359 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:53,359 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:56:53,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 17:56:53,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 17:56:53,771 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:56:53,776 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5c47f717ab27776f2d38f585a39f2180, NAME => 'TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp 2023-04-06 17:56:53,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 17:56:54,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:54,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5c47f717ab27776f2d38f585a39f2180, disabling compactions & flushes 2023-04-06 17:56:54,191 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. after waiting 0 ms 2023-04-06 17:56:54,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,191 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,191 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:56:54,198 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 17:56:54,202 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803814201"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803814201"}]},"ts":"1680803814201"} 2023-04-06 17:56:54,204 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 17:56:54,206 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 17:56:54,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803814206"}]},"ts":"1680803814206"} 2023-04-06 17:56:54,208 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 17:56:54,212 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, ASSIGN}] 2023-04-06 17:56:54,214 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, ASSIGN 2023-04-06 17:56:54,216 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=false 2023-04-06 17:56:54,367 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:54,368 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803814367"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803814367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803814367"}]},"ts":"1680803814367"} 2023-04-06 17:56:54,371 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:56:54,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 17:56:54,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5c47f717ab27776f2d38f585a39f2180, NAME => 'TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:56:54,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:54,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,530 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,532 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A 2023-04-06 17:56:54,532 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A 2023-04-06 17:56:54,532 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:54,532 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c47f717ab27776f2d38f585a39f2180 columnFamilyName A 2023-04-06 17:56:54,533 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(310): Store=5c47f717ab27776f2d38f585a39f2180/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:54,533 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,534 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B 2023-04-06 17:56:54,534 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B 2023-04-06 17:56:54,534 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:54,534 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c47f717ab27776f2d38f585a39f2180 columnFamilyName B 2023-04-06 17:56:54,535 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(310): Store=5c47f717ab27776f2d38f585a39f2180/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:54,535 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,536 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C 2023-04-06 17:56:54,536 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C 2023-04-06 17:56:54,536 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:54,536 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c47f717ab27776f2d38f585a39f2180 columnFamilyName C 2023-04-06 17:56:54,537 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(310): Store=5c47f717ab27776f2d38f585a39f2180/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:54,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:56:54,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:54,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:56:54,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 5c47f717ab27776f2d38f585a39f2180; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65629580, jitterRate=-0.022043049335479736}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:56:54,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:56:54,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., pid=18, masterSystemTime=1680803814524 2023-04-06 17:56:54,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:54,545 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:54,545 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803814545"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803814545"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803814545"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803814545"}]},"ts":"1680803814545"} 2023-04-06 17:56:54,548 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-06 17:56:54,548 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 in 175 msec 2023-04-06 17:56:54,549 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-06 17:56:54,549 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, ASSIGN in 337 msec 2023-04-06 17:56:54,550 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 17:56:54,550 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803814550"}]},"ts":"1680803814550"} 2023-04-06 17:56:54,551 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 17:56:54,553 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 17:56:54,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2000 sec 2023-04-06 17:56:55,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 17:56:55,469 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-06 17:56:55,471 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3efe2ee2 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:55,478 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@298b7d18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:55,480 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:55,485 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:55,487 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 17:56:55,492 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 17:56:55,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:56:55,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:56:55,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 17:56:55,919 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 17:56:55,921 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:56:55,921 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 17:56:55,927 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 17:56:55,939 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, REOPEN/MOVE}] 2023-04-06 17:56:55,940 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, REOPEN/MOVE 2023-04-06 17:56:55,941 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:55,941 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803815941"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803815941"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803815941"}]},"ts":"1680803815941"} 2023-04-06 17:56:55,942 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:56:56,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 5c47f717ab27776f2d38f585a39f2180, disabling compactions & flushes 2023-04-06 17:56:56,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. after waiting 0 ms 2023-04-06 17:56:56,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,101 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 17:56:56,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,102 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:56:56,102 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5c47f717ab27776f2d38f585a39f2180 to self. 2023-04-06 17:56:56,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,105 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=CLOSED 2023-04-06 17:56:56,105 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803816105"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803816105"}]},"ts":"1680803816105"} 2023-04-06 17:56:56,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-06 17:56:56,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 in 165 msec 2023-04-06 17:56:56,110 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=true 2023-04-06 17:56:56,260 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,261 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803816260"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803816260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803816260"}]},"ts":"1680803816260"} 2023-04-06 17:56:56,265 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:56:56,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5c47f717ab27776f2d38f585a39f2180, NAME => 'TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:56:56,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:56:56,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,428 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,429 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A 2023-04-06 17:56:56,429 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A 2023-04-06 17:56:56,429 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:56,433 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c47f717ab27776f2d38f585a39f2180 columnFamilyName A 2023-04-06 17:56:56,436 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(310): Store=5c47f717ab27776f2d38f585a39f2180/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:56,437 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,438 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B 2023-04-06 17:56:56,438 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B 2023-04-06 17:56:56,438 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:56,438 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c47f717ab27776f2d38f585a39f2180 columnFamilyName B 2023-04-06 17:56:56,439 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(310): Store=5c47f717ab27776f2d38f585a39f2180/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:56,439 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,440 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C 2023-04-06 17:56:56,440 DEBUG [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C 2023-04-06 17:56:56,440 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:56:56,440 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c47f717ab27776f2d38f585a39f2180 columnFamilyName C 2023-04-06 17:56:56,441 INFO [StoreOpener-5c47f717ab27776f2d38f585a39f2180-1] regionserver.HStore(310): Store=5c47f717ab27776f2d38f585a39f2180/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:56:56,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:56:56,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 5c47f717ab27776f2d38f585a39f2180; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66644367, jitterRate=-0.006921544671058655}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:56:56,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:56:56,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., pid=23, masterSystemTime=1680803816419 2023-04-06 17:56:56,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,449 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,449 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803816449"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803816449"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803816449"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803816449"}]},"ts":"1680803816449"} 2023-04-06 17:56:56,452 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-06 17:56:56,452 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 in 185 msec 2023-04-06 17:56:56,454 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-06 17:56:56,454 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, REOPEN/MOVE in 513 msec 2023-04-06 17:56:56,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-06 17:56:56,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 531 msec 2023-04-06 17:56:56,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 956 msec 2023-04-06 17:56:56,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-06 17:56:56,467 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x13522ca3 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,471 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b0388c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,472 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x5499dabb to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,475 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3584fc9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,475 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x6565606b to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,478 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f33227e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,479 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x65dbf2b7 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,484 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31d0ca0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,484 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x537a0ff8 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,488 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@698e116a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,488 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3e8e6c to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,493 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6322d500, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,493 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x31c99ac6 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,497 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f8a3f7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,498 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x4e7aae9f to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,503 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3318123c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,504 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3ca9d9a3 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:56:56,509 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@273c86ff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:56:56,516 DEBUG [hconnection-0x93aac27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,516 DEBUG [hconnection-0x26b63fe2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,517 DEBUG [hconnection-0x2cc9af44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,517 DEBUG [hconnection-0xfbc83da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,518 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,519 DEBUG [hconnection-0x13bc4f62-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,519 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57630, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,521 DEBUG [hconnection-0x4fe06f13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,522 DEBUG [hconnection-0x9f25510-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,522 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,523 DEBUG [hconnection-0x5845f0ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,522 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,523 DEBUG [hconnection-0x316a72c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:56:56,524 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:56,526 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,529 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,529 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,531 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:56:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:56,545 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:56:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:56,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:56,547 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:56,547 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:56,547 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:56,547 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:56,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:56:56,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:56,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:56:56,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:56,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:56:56,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:56,550 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,551 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:56,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:56,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:56,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:56,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:56,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:56,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:56,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:56,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803876571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803876573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803816566, max timestamp=1680803816572Num uniques 859; 2023-04-06 17:56:56,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:56,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803816566, max timestamp=1680803816572Num uniques 859; 2023-04-06 17:56:56,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:56,580 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,580 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:56,580 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:56,580 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:56,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:56,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:56,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:56,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803816566, max timestamp=1680803816577Num uniques 859; 2023-04-06 17:56:56,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:56:56,586 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2465f30e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:56,588 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:56,588 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,589 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:56,589 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:56,589 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:56,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:56,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:56,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:56,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:56,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:56,590 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:56,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:56,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:56,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:56,590 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:56,592 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,592 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:56,592 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:56,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803876597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803876598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803876598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:56,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:56,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:56,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:56,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:56,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:56,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:56,607 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:56,607 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:56,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:56,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803876678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803876678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803876700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803876700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,707 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:56,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803876709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:56,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:56,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:56,718 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:56,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:56,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:56,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:56,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,719 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:56,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:56,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:56,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:56,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:56,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:56,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:56,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:56,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:56,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:56,724 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,724 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:56,724 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:56,724 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:56,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:56,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,727 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:56,733 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:56,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704a7cc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:56,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:56,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:56,748 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:56,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:56,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:56,751 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:56,751 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:56,751 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:56,751 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:56,751 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:56,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:56,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:56,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:56,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:56,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:56,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:56,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:56,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:56,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,764 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:56,764 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:56,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:56,864 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:56,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:56,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:56,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:56,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:56,869 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:56,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:56,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:56,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:56,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:56,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:56,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:56,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803876883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803876883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:56,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:56,886 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:56,886 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:56,886 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:56,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:56,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:56,892 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,892 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:56,892 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:56,892 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:56,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:56,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:56,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:56,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:56,893 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,893 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,894 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,894 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:56,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bdd40cf[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:56,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:56,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,896 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,896 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:56,896 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:56,897 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:56,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:56,897 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:56,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:56,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:56,900 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:56,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:56,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:56,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:56,900 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:56,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:56,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,902 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:56,903 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,905 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:56,905 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:56,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:56,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:56,906 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:56,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:56,906 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:56,906 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,906 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803876906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,906 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:56,906 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803876906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,906 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:56,907 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,907 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:56,907 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:56,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:56,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803876912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,006 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,021 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304067b568af4d3b744dab242932276fe1732 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067b568af4d3b744dab242932276fe1732 2023-04-06 17:56:57,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,030 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,031 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,048 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,048 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,048 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,048 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,049 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:57,049 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,051 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:57,052 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:57,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:57,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aebf11e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:57,055 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:57,055 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,056 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,056 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:57,056 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,056 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:57,056 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:57,056 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:57,057 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:57,057 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,057 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,057 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:57,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:57,061 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:57,061 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:57,061 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:57,061 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:57,061 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:57,063 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,063 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,063 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,063 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:57,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,072 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:57,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,073 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:57,073 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:57,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,173 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,183 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,183 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,183 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,183 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,184 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,184 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,184 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,185 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,185 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803877185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:57,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:57,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:57,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803877186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,189 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:57,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@211d8ace[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:57,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:57,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,192 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,192 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:57,192 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:57,192 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:57,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:57,193 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:57,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:57,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:57,197 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:57,197 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:57,197 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:57,197 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:57,197 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:57,198 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,199 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,199 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,204 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:57,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:57,205 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:57,205 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803877209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803877209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803877216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,305 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,308 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,311 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,311 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,311 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:57,311 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:57,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:57,312 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:57,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:57,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30573e6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:57,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,315 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:57,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:57,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:57,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:57,315 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,315 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:57,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:57,317 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:57,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:57,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:57,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:57,318 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:57,318 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,318 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,318 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,320 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:57,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:57,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,322 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,322 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,322 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,422 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,426 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,427 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,427 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,427 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,427 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,428 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,428 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,429 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:57,429 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:57,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:57,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:57,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@227f9071[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:57,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,432 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:57,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:57,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:57,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:57,433 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:57,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,433 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,433 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:57,435 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:57,435 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:57,435 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:57,436 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:57,436 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:57,436 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:57,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:57,441 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,441 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:57,441 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:57,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:57,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,445 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/687f0f570e004332ad5bc68432cc72cc 2023-04-06 17:56:57,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/26e7ad86920547fb853acaf7a83df506 2023-04-06 17:56:57,541 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,545 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:57,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:57,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:57,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:57,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:57,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:57,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5769e621[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:57,550 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:57,550 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,550 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,550 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:57,551 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:57,551 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:57,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:57,551 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,551 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,551 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,551 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:57,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:57,553 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:57,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:57,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:57,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:57,553 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:57,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,555 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,556 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:57,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:57,558 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,558 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,558 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,658 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,662 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,669 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,669 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,669 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,676 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,677 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,677 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:57,677 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:57,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:57,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:57,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:57,682 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,682 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,682 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,682 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:57,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aef4df8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:57,683 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:57,684 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,684 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,684 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,684 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:57,686 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:57,686 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:57,686 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:57,687 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:57,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,687 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,687 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:57,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:57,689 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:57,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:57,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:57,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:57,689 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:57,690 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,690 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803877689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,690 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,691 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803877691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:57,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:57,694 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,694 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:57,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803877712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803877714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:57,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803877720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,794 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,797 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,798 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,798 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,798 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,798 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,799 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,799 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,800 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,800 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,800 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:57,801 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,801 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:57,801 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:57,801 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,801 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:57,801 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:57,801 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,801 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,802 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,802 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:57,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325d9700[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:57,803 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:57,803 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,803 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:57,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:57,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:57,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:57,804 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:57,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:57,807 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:57,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:57,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:57,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:57,807 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:57,809 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,810 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,810 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:57,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:57,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,819 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:57,823 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,823 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:57,823 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:57,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:57,824 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:57,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/066af0f2852048b1847f897587ff08d2 2023-04-06 17:56:57,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/687f0f570e004332ad5bc68432cc72cc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/687f0f570e004332ad5bc68432cc72cc 2023-04-06 17:56:57,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/687f0f570e004332ad5bc68432cc72cc, entries=200, sequenceid=16, filesize=31.9 K 2023-04-06 17:56:57,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/26e7ad86920547fb853acaf7a83df506 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/26e7ad86920547fb853acaf7a83df506 2023-04-06 17:56:57,892 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 17:56:57,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/26e7ad86920547fb853acaf7a83df506, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 17:56:57,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/066af0f2852048b1847f897587ff08d2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/066af0f2852048b1847f897587ff08d2 2023-04-06 17:56:57,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/066af0f2852048b1847f897587ff08d2, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 17:56:57,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.08 KB/130125 for 5c47f717ab27776f2d38f585a39f2180 in 1357ms, sequenceid=16, compaction requested=false 2023-04-06 17:56:57,902 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-06 17:56:57,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:56:57,924 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:57,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:57,931 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:57,931 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,931 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:57,931 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:57,931 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,932 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,932 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:57,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:57,933 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:57,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:57,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:57,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:57,934 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:57,934 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:57,934 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:57,935 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:57,935 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:57,942 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:57,942 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:57,942 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:57,942 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:57,942 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=127.08 KB heapSize=364.15 KB 2023-04-06 17:56:57,943 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:56:57,943 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:57,943 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:56:57,943 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:57,943 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:56:57,943 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:57,978 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406e9e5e2c2885a43e4bc1a5eac791126b3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e9e5e2c2885a43e4bc1a5eac791126b3 2023-04-06 17:56:57,996 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/9c09a9830beb44bcbd72047a841790fb 2023-04-06 17:56:58,413 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/9dcf67b54bd64abd9c0308079afcb654 2023-04-06 17:56:58,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:58,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:58,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:58,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803878710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:58,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803878713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:58,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803878719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:58,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803878722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:58,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803878726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:58,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803878815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:58,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803878817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,825 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/4473a68fcfb34a27b1adca610c543f98 2023-04-06 17:56:58,830 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/9c09a9830beb44bcbd72047a841790fb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/9c09a9830beb44bcbd72047a841790fb 2023-04-06 17:56:58,833 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/9c09a9830beb44bcbd72047a841790fb, entries=150, sequenceid=52, filesize=25.1 K 2023-04-06 17:56:58,834 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/9dcf67b54bd64abd9c0308079afcb654 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9dcf67b54bd64abd9c0308079afcb654 2023-04-06 17:56:58,838 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9dcf67b54bd64abd9c0308079afcb654, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 17:56:58,839 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/4473a68fcfb34a27b1adca610c543f98 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4473a68fcfb34a27b1adca610c543f98 2023-04-06 17:56:58,844 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4473a68fcfb34a27b1adca610c543f98, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 17:56:58,845 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.08 KB/130125, heapSize ~364.10 KB/372840, currentSize=53.67 KB/54960 for 5c47f717ab27776f2d38f585a39f2180 in 903ms, sequenceid=52, compaction requested=false 2023-04-06 17:56:58,845 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:56:58,845 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:58,849 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:58,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:58,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:58,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cff7522[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:58,851 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:58,851 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,851 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:58,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:58,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:58,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:58,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:58,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:58,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:58,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:58,856 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:58,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:58,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:58,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:58,856 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:58,863 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,863 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,863 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:58,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:58,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:58,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:58,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:58,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:58,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:58,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:58,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:58,868 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:58,868 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:58,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:58,968 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:58,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:58,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:58,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:58,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:58,973 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:58,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:58,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:58,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:58,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:58,974 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:58,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:58,974 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:58,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:58,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:58,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:58,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:58,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:58,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:58,975 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:58,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:58,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:58,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:58,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:58,976 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:58,976 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:58,976 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:56:58,976 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:56:58,976 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:58,976 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:56:58,977 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:58,977 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:56:58,977 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:56:59,003 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040649e6ff62c7d3410291ce27842e1ba9d3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040649e6ff62c7d3410291ce27842e1ba9d3 2023-04-06 17:56:59,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:56:59,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:56:59,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803819037, max timestamp=1680803819047Num uniques 859; 2023-04-06 17:56:59,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803879048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:56:59,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803819039, max timestamp=1680803819047Num uniques 859; 2023-04-06 17:56:59,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803819037, max timestamp=1680803819047Num uniques 859; 2023-04-06 17:56:59,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:56:59,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:56:59,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803879100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803879150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803879205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803879354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,409 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/a4960e91d22c4c63a1bee86bd46f59b6 2023-04-06 17:56:59,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803879409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803879658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:56:59,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803879715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,819 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/a25bd9e03a8a4f14b85080aaa7bc0b26 2023-04-06 17:56:59,830 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/5c21f0ca9ac94fe9a6d13db842307852 2023-04-06 17:56:59,834 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/a4960e91d22c4c63a1bee86bd46f59b6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/a4960e91d22c4c63a1bee86bd46f59b6 2023-04-06 17:56:59,838 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/a4960e91d22c4c63a1bee86bd46f59b6, entries=150, sequenceid=63, filesize=25.1 K 2023-04-06 17:56:59,839 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/a25bd9e03a8a4f14b85080aaa7bc0b26 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/a25bd9e03a8a4f14b85080aaa7bc0b26 2023-04-06 17:56:59,842 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/a25bd9e03a8a4f14b85080aaa7bc0b26, entries=150, sequenceid=63, filesize=11.6 K 2023-04-06 17:56:59,843 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/5c21f0ca9ac94fe9a6d13db842307852 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/5c21f0ca9ac94fe9a6d13db842307852 2023-04-06 17:56:59,847 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/5c21f0ca9ac94fe9a6d13db842307852, entries=150, sequenceid=63, filesize=11.6 K 2023-04-06 17:56:59,848 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 5c47f717ab27776f2d38f585a39f2180 in 872ms, sequenceid=63, compaction requested=true 2023-04-06 17:56:59,848 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:56:59,848 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:56:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:56:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:56:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:56:59,849 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:56:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:59,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:56:59,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743fce72[Count = 0] remaining members to acquire global barrier 2023-04-06 17:56:59,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:56:59,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:56:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:56:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:56:59,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:56:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:56:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:59,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:56:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:56:59,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:56:59,854 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:56:59,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:56:59,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:56:59,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:56:59,854 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:56:59,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:56:59,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:56:59,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:59,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,857 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,858 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:59,858 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:56:59,858 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:59,858 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:56:59,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:56:59,858 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:59,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:56:59,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:56:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:59,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:59,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:56:59,859 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:56:59,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,859 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:56:59,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:56:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:56:59,959 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:56:59,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:56:59,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:56:59,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:56:59,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:56:59,962 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:56:59,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:59,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:56:59,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:56:59,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:59,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:56:59,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:56:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:56:59,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:56:59,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:56:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:59,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:56:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:56:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:56:59,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:56:59,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:56:59,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:56:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:56:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:56:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:56:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:56:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:56:59,964 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:56:59,965 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:56:59,965 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-06 17:56:59,965 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:56:59,965 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:59,965 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:56:59,965 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:59,965 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:56:59,965 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:56:59,989 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069545ed7bd46c462c80c6dabb140cffc2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069545ed7bd46c462c80c6dabb140cffc2 2023-04-06 17:57:00,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:00,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:00,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803880190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803880221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803880296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,403 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/17a9ca0dea814f19b6d7b661bffc101d 2023-04-06 17:57:00,418 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/2dcd4bd4b6754af3ad46044680fce1ea 2023-04-06 17:57:00,431 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/8b8f473920a64f849d07507bc95d904f 2023-04-06 17:57:00,435 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/17a9ca0dea814f19b6d7b661bffc101d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/17a9ca0dea814f19b6d7b661bffc101d 2023-04-06 17:57:00,439 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/17a9ca0dea814f19b6d7b661bffc101d, entries=150, sequenceid=98, filesize=25.1 K 2023-04-06 17:57:00,440 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/2dcd4bd4b6754af3ad46044680fce1ea as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2dcd4bd4b6754af3ad46044680fce1ea 2023-04-06 17:57:00,445 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2dcd4bd4b6754af3ad46044680fce1ea, entries=150, sequenceid=98, filesize=11.6 K 2023-04-06 17:57:00,450 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/8b8f473920a64f849d07507bc95d904f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/8b8f473920a64f849d07507bc95d904f 2023-04-06 17:57:00,454 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/8b8f473920a64f849d07507bc95d904f, entries=150, sequenceid=98, filesize=11.6 K 2023-04-06 17:57:00,456 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for 5c47f717ab27776f2d38f585a39f2180 in 491ms, sequenceid=98, compaction requested=true 2023-04-06 17:57:00,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:00,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,456 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:00,456 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:00,456 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:00,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:00,459 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,459 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:00,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14dfdd80[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:00,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:00,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,461 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:00,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:00,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:00,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:00,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:00,462 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:00,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:00,465 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:00,465 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:00,465 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:00,465 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:00,465 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:00,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:00,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,480 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,481 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:00,481 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:00,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:00,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:57:00,502 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:00,502 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:00,502 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:00,502 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:00,502 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:00,502 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:00,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803880550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,581 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:00,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:00,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:00,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:00,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:00,584 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:00,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:00,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:00,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,585 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,585 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,585 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:00,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:00,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:00,586 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,586 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:00,586 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:00,586 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:00,587 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,588 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,588 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:00,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:00,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae4bd8c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:00,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:00,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:00,590 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,591 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:00,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:00,592 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:00,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:00,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:00,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:00,592 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:00,593 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,593 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,593 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:00,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:00,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:00,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,606 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:00,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,606 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:00,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803880655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,706 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:00,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:00,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:00,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:00,709 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:00,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:00,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:00,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,710 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,710 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,710 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,710 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:00,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:00,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:00,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:00,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:00,712 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,712 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:00,712 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:00,712 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:00,713 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:00,714 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:00,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d258797[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:00,714 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,715 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:00,715 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,715 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:00,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:00,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:00,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,716 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,716 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,716 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:00,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:00,718 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:00,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:00,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:00,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:00,718 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:00,719 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,719 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,719 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:00,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:00,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:00,722 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,723 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:00,723 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:00,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803880728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803880728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,731 DEBUG [Thread-947] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:00,731 DEBUG [Thread-951] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:00,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803880733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,736 DEBUG [Thread-953] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:00,823 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:00,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:00,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:00,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:00,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:00,826 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:00,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:00,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:00,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,827 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,827 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,827 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,827 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,827 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:00,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:00,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:00,828 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:00,828 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,828 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:00,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:00,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:00,829 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,829 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:00,829 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:00,829 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:00,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:00,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:00,829 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:00,830 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:00,831 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:00,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57638e1f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:00,831 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:00,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:00,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:00,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:00,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,833 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:00,833 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,833 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:00,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:00,835 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:00,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:00,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:00,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:00,835 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:00,835 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,835 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:00,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:00,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:00,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,839 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:00,839 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:00,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:00,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803880858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,914 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406054886b3e6b54bacba00676f8a3f7fb9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406054886b3e6b54bacba00676f8a3f7fb9 2023-04-06 17:57:00,939 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:00,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:00,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:00,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:00,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:00,942 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:00,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:00,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:00,943 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,943 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,943 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,943 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,944 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,944 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:00,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:00,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:00,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:00,946 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,946 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,946 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,946 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:00,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3a7aae[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:00,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:00,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,951 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:00,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:00,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:00,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:00,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:00,952 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:00,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:00,954 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:00,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:00,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:00,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:00,954 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:00,955 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,955 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:00,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:00,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,957 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:00,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:00,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:00,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:00,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:00,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:00,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:00,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:00,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:00,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:00,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:00,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:00,960 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:00,960 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,060 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,065 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,065 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,066 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,067 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,067 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,067 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,067 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,068 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525db858[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,070 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,071 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,073 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,073 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,077 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:01,077 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,077 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:01,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803881161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,177 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,181 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,182 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,183 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,183 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,183 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,183 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,184 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,184 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,184 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,185 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,185 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b584bb[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,185 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,186 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,187 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,188 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,189 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,189 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,189 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,189 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,198 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,213 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,213 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,213 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,214 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:01,214 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,214 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,214 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,214 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,214 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,214 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,214 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:01,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803881226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,314 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,317 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,318 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,319 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/ac94daa7fda34233a5168fd48ab8e2db 2023-04-06 17:57:01,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,320 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,320 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,320 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,320 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,321 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,321 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,321 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686de5a4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,334 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,334 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,334 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,338 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,341 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,341 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,348 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,348 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,348 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:01,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,356 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,356 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,456 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,459 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,461 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,464 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,465 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,465 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec1d0d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,488 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,488 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,488 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,488 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,488 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,492 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:01,492 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,493 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,593 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,602 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,603 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394ceaf0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,609 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,611 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,611 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,611 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,611 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,611 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,616 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,616 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:01,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803881664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,716 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,719 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,720 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,721 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,721 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,722 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,722 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3653313e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,725 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,726 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,726 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,726 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,728 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,728 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,729 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,729 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,729 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,731 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,733 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,733 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,733 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:01,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,733 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,733 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/29e70b1fc9d5453dba4b0341163888b1 2023-04-06 17:57:01,833 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,837 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,839 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3194b295[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,843 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,843 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,843 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,846 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,846 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,847 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,851 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,851 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,951 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:01,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:01,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:01,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:01,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:01,954 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:01,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:01,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:01,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,955 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:01,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:01,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:01,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:01,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:01,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:01,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:01,958 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,958 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,958 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,958 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:01,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:01,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5776d5df[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:01,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:01,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:01,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:01,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:01,961 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:01,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:01,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:01,962 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:01,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:01,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:01,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:01,962 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:01,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:01,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:01,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:01,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:01,965 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:01,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:01,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:01,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:01,966 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:01,966 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:01,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:01,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:02,066 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:02,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:02,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:02,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:02,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:02,069 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:02,069 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:02,069 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:02,069 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:02,069 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,069 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,070 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:02,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:02,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:02,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:02,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:02,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:02,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:02,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:02,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:02,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:02,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:02,072 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:02,072 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:02,072 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:02,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:02,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:02,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:02,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:02,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:02,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:02,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:02,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b3aeab[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:02,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:02,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,075 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,075 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:02,075 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:02,075 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:02,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:02,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:02,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:02,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:02,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:02,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:02,078 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:02,078 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:02,078 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:02,078 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:02,078 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:02,079 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,079 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,079 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:02,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:02,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:02,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:02,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:02,083 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:02,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:02,083 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:02,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:02,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:02,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/103bd906bab2496cbd0470e4d3e0e696 2023-04-06 17:57:02,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/ac94daa7fda34233a5168fd48ab8e2db as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/ac94daa7fda34233a5168fd48ab8e2db 2023-04-06 17:57:02,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/ac94daa7fda34233a5168fd48ab8e2db, entries=200, sequenceid=110, filesize=31.9 K 2023-04-06 17:57:02,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/29e70b1fc9d5453dba4b0341163888b1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/29e70b1fc9d5453dba4b0341163888b1 2023-04-06 17:57:02,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/29e70b1fc9d5453dba4b0341163888b1, entries=150, sequenceid=110, filesize=11.6 K 2023-04-06 17:57:02,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/103bd906bab2496cbd0470e4d3e0e696 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/103bd906bab2496cbd0470e4d3e0e696 2023-04-06 17:57:02,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/103bd906bab2496cbd0470e4d3e0e696, entries=150, sequenceid=110, filesize=11.6 K 2023-04-06 17:57:02,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5c47f717ab27776f2d38f585a39f2180 in 1676ms, sequenceid=110, compaction requested=true 2023-04-06 17:57:02,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:02,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:02,177 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:57:02,177 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:57:02,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:02,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:02,179 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 142610 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:57:02,179 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/A is initiating minor compaction (all files) 2023-04-06 17:57:02,179 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:57:02,179 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/A in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:02,179 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/B is initiating minor compaction (all files) 2023-04-06 17:57:02,179 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/687f0f570e004332ad5bc68432cc72cc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/9c09a9830beb44bcbd72047a841790fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/a4960e91d22c4c63a1bee86bd46f59b6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/17a9ca0dea814f19b6d7b661bffc101d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/ac94daa7fda34233a5168fd48ab8e2db] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=139.3 K 2023-04-06 17:57:02,179 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/B in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:02,179 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/26e7ad86920547fb853acaf7a83df506, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9dcf67b54bd64abd9c0308079afcb654, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/a25bd9e03a8a4f14b85080aaa7bc0b26, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2dcd4bd4b6754af3ad46044680fce1ea, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/29e70b1fc9d5453dba4b0341163888b1] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=58.0 K 2023-04-06 17:57:02,179 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 687f0f570e004332ad5bc68432cc72cc, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680803816538 2023-04-06 17:57:02,179 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 26e7ad86920547fb853acaf7a83df506, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680803816541 2023-04-06 17:57:02,180 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9dcf67b54bd64abd9c0308079afcb654, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680803816590 2023-04-06 17:57:02,180 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 9c09a9830beb44bcbd72047a841790fb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680803816590 2023-04-06 17:57:02,180 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a25bd9e03a8a4f14b85080aaa7bc0b26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680803818703 2023-04-06 17:57:02,180 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting a4960e91d22c4c63a1bee86bd46f59b6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680803818703 2023-04-06 17:57:02,181 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2dcd4bd4b6754af3ad46044680fce1ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803819092 2023-04-06 17:57:02,181 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 17a9ca0dea814f19b6d7b661bffc101d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803819092 2023-04-06 17:57:02,181 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 29e70b1fc9d5453dba4b0341163888b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680803820177 2023-04-06 17:57:02,181 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ac94daa7fda34233a5168fd48ab8e2db, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680803820177 2023-04-06 17:57:02,183 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:02,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:02,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:02,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:02,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:02,187 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:02,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:02,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:02,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:02,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,188 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:02,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:02,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:02,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:02,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:02,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:02,190 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:02,190 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:02,190 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:57:02,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:02,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:02,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:02,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:02,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:02,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:02,196 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#A#compaction#138 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:02,223 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#B#compaction#140 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:02,227 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068d8255d3fd774c91b398f929e2e96874 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068d8255d3fd774c91b398f929e2e96874 2023-04-06 17:57:02,237 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/d24be39dd5bb4563b6759ed2bd26a1fa as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d24be39dd5bb4563b6759ed2bd26a1fa 2023-04-06 17:57:02,244 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/B of 5c47f717ab27776f2d38f585a39f2180 into d24be39dd5bb4563b6759ed2bd26a1fa(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:02,244 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:02,244 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/B, priority=11, startTime=1680803822177; duration=0sec 2023-04-06 17:57:02,245 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:02,245 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:57:02,247 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:57:02,247 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/C is initiating minor compaction (all files) 2023-04-06 17:57:02,248 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/C in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:02,248 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/066af0f2852048b1847f897587ff08d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4473a68fcfb34a27b1adca610c543f98, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/5c21f0ca9ac94fe9a6d13db842307852, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/8b8f473920a64f849d07507bc95d904f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/103bd906bab2496cbd0470e4d3e0e696] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=58.0 K 2023-04-06 17:57:02,249 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 066af0f2852048b1847f897587ff08d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680803816541 2023-04-06 17:57:02,251 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4473a68fcfb34a27b1adca610c543f98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680803816590 2023-04-06 17:57:02,252 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5c21f0ca9ac94fe9a6d13db842307852, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680803818703 2023-04-06 17:57:02,256 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 8b8f473920a64f849d07507bc95d904f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680803819092 2023-04-06 17:57:02,256 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 103bd906bab2496cbd0470e4d3e0e696, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680803820177 2023-04-06 17:57:02,280 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#C#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:02,290 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/2db9d70140534c038d42c3f8dc744717 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/2db9d70140534c038d42c3f8dc744717 2023-04-06 17:57:02,296 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/C of 5c47f717ab27776f2d38f585a39f2180 into 2db9d70140534c038d42c3f8dc744717(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:02,296 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:02,296 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/C, priority=11, startTime=1680803822177; duration=0sec 2023-04-06 17:57:02,296 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:02,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:02,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:02,673 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/782e8a4d492f44648aa5b1a86983da00 2023-04-06 17:57:02,690 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/b2d76d3843444befbe7265a1d50a276a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b2d76d3843444befbe7265a1d50a276a 2023-04-06 17:57:02,690 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/dc248fb07e374d478fdb9eb1b105ccfa 2023-04-06 17:57:02,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:02,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803882697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:02,701 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/A of 5c47f717ab27776f2d38f585a39f2180 into b2d76d3843444befbe7265a1d50a276a(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:02,701 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:02,701 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/A, priority=11, startTime=1680803822177; duration=0sec 2023-04-06 17:57:02,702 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:02,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:02,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803882798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:03,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803882999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,121 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/1cfd6c957caf4fa58a37f8142f645151 2023-04-06 17:57:03,127 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/782e8a4d492f44648aa5b1a86983da00 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/782e8a4d492f44648aa5b1a86983da00 2023-04-06 17:57:03,132 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/782e8a4d492f44648aa5b1a86983da00, entries=150, sequenceid=130, filesize=25.2 K 2023-04-06 17:57:03,133 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/dc248fb07e374d478fdb9eb1b105ccfa as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dc248fb07e374d478fdb9eb1b105ccfa 2023-04-06 17:57:03,141 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dc248fb07e374d478fdb9eb1b105ccfa, entries=150, sequenceid=130, filesize=11.7 K 2023-04-06 17:57:03,142 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/1cfd6c957caf4fa58a37f8142f645151 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/1cfd6c957caf4fa58a37f8142f645151 2023-04-06 17:57:03,147 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/1cfd6c957caf4fa58a37f8142f645151, entries=150, sequenceid=130, filesize=11.7 K 2023-04-06 17:57:03,148 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5c47f717ab27776f2d38f585a39f2180 in 958ms, sequenceid=130, compaction requested=false 2023-04-06 17:57:03,148 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:03,148 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,148 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,148 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,148 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,148 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,148 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26c6a462[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,153 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,153 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,153 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,153 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,156 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,156 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,156 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,156 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,156 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,159 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,162 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,162 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,162 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,162 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:03,235 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:57:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:03,252 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406dcec363139f64aedb98b6b03f9f03b2e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406dcec363139f64aedb98b6b03f9f03b2e 2023-04-06 17:57:03,262 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:03,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:03,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:03,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:03,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:03,265 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:03,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:03,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:03,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,266 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,266 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,266 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,266 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:03,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:03,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:03,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:03,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:03,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:03,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:03,267 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,267 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:03,267 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:03,267 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d2ba35[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,270 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,270 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,271 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,271 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,271 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,271 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,274 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,274 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,276 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,277 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,277 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,277 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,278 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,278 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,278 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:03,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803883283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:03,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803883303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,378 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:03,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:03,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:03,381 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:03,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:03,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:03,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:03,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:03,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803883384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:03,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:03,385 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:03,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:03,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53eb499d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,389 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,391 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,393 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,393 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,394 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,394 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,394 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,394 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,397 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,398 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,398 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,498 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:03,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:03,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:03,500 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:03,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:03,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:03,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,501 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,502 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,502 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:03,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:03,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:03,503 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,503 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:03,503 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:03,503 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,504 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,504 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,504 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,504 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473963bd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,506 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,507 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,507 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,507 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,509 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,509 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,510 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,510 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,510 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,510 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,511 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:03,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,514 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,514 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,514 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,514 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803883587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,614 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:03,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:03,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:03,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:03,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:03,617 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:03,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:03,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:03,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:03,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:03,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:03,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:03,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:03,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:03,619 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,619 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:03,619 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:03,620 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54cf2a8c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,622 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,622 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,622 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,622 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,622 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,624 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,624 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,624 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,624 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,626 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,626 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,628 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,629 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,630 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:03,631 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,631 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,631 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,631 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,631 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,631 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,657 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/e4e6fe65452a46d5970f8a4e105c2aa6 2023-04-06 17:57:03,731 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:03,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:03,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:03,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:03,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:03,734 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:03,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:03,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:03,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:03,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:03,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:03,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:03,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,736 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,737 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736b2132[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,748 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,750 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,751 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,752 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,752 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,752 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,755 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,755 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,755 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:03,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803883808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,855 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:03,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:03,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:03,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:03,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:03,858 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:03,858 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,858 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:03,858 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:03,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,860 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,860 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:03,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:03,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:03,860 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:03,860 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,860 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:03,861 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,861 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:03,861 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:03,861 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,862 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,863 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a452963[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,863 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,865 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,865 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,865 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,865 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,867 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,867 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,871 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,871 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:03,871 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803883890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,971 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:03,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:03,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:03,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:03,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:03,974 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:03,974 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,974 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:03,974 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:03,974 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,974 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,975 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:03,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:03,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:03,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:03,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:03,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:03,976 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:03,977 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,977 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:03,977 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:03,977 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:03,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:03,979 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:03,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766d991d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:03,979 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,980 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,980 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:03,980 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:03,980 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:03,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:03,980 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,980 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:03,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:03,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:03,983 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:03,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:03,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:03,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:03,983 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:03,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:03,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:03,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,987 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:03,987 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:03,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:03,987 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:03,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:03,987 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,987 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:03,987 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:03,987 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:03,988 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:03,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:03,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:03,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:04,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/30fe055e74d64b2d95d45009c64a31ab 2023-04-06 17:57:04,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/ff35fc0e180c4b408e67c0adb1e287aa 2023-04-06 17:57:04,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/e4e6fe65452a46d5970f8a4e105c2aa6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e4e6fe65452a46d5970f8a4e105c2aa6 2023-04-06 17:57:04,088 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:04,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:04,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e4e6fe65452a46d5970f8a4e105c2aa6, entries=200, sequenceid=146, filesize=32.1 K 2023-04-06 17:57:04,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/30fe055e74d64b2d95d45009c64a31ab as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30fe055e74d64b2d95d45009c64a31ab 2023-04-06 17:57:04,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:04,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:04,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:04,092 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:04,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:04,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:04,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:04,093 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,093 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,094 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,094 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:04,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:04,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:04,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:04,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30fe055e74d64b2d95d45009c64a31ab, entries=150, sequenceid=146, filesize=11.8 K 2023-04-06 17:57:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:04,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:04,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:04,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:04,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:04,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:04,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:04,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:04,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:04,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:04,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:04,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:04,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:04,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:04,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:04,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:04,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/ff35fc0e180c4b408e67c0adb1e287aa as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/ff35fc0e180c4b408e67c0adb1e287aa 2023-04-06 17:57:04,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/ff35fc0e180c4b408e67c0adb1e287aa, entries=150, sequenceid=146, filesize=11.8 K 2023-04-06 17:57:04,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5c47f717ab27776f2d38f585a39f2180 in 872ms, sequenceid=146, compaction requested=true 2023-04-06 17:57:04,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:04,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:04,107 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:04,107 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:04,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:04,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:04,108 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36030 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:04,108 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84600 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:04,108 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/B is initiating minor compaction (all files) 2023-04-06 17:57:04,108 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/A is initiating minor compaction (all files) 2023-04-06 17:57:04,108 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/B in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:04,108 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/A in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:04,108 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d24be39dd5bb4563b6759ed2bd26a1fa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dc248fb07e374d478fdb9eb1b105ccfa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30fe055e74d64b2d95d45009c64a31ab] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=35.2 K 2023-04-06 17:57:04,108 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b2d76d3843444befbe7265a1d50a276a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/782e8a4d492f44648aa5b1a86983da00, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e4e6fe65452a46d5970f8a4e105c2aa6] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=82.6 K 2023-04-06 17:57:04,109 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d24be39dd5bb4563b6759ed2bd26a1fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680803820177 2023-04-06 17:57:04,109 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b2d76d3843444befbe7265a1d50a276a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680803820177 2023-04-06 17:57:04,110 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting dc248fb07e374d478fdb9eb1b105ccfa, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680803820543 2023-04-06 17:57:04,110 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 782e8a4d492f44648aa5b1a86983da00, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680803820543 2023-04-06 17:57:04,110 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 30fe055e74d64b2d95d45009c64a31ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680803822695 2023-04-06 17:57:04,110 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting e4e6fe65452a46d5970f8a4e105c2aa6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680803822688 2023-04-06 17:57:04,115 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#B#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:04,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:04,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:04,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:04,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5b5616[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:04,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:04,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:04,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:04,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:04,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:04,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:04,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:04,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:04,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,124 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#A#compaction#148 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:04,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:04,126 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:04,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:04,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:04,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:04,126 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:04,127 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,127 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,127 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:04,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,129 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:04,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:04,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:04,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:04,131 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:04,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:04,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:04,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:04,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,131 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:04,131 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:04,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:04,231 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:04,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:04,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:04,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:04,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:04,234 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:04,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:04,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:04,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:04,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:04,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:04,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:04,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:04,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:04,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:04,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:04,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:04,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:04,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:04,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:04,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:04,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:04,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:04,242 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:57:04,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:04,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:04,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:04,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:04,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:04,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:04,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:04,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:04,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:04,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803884425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:04,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803884530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,540 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/3a0096183aa34b57837e2e5ac874479a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/3a0096183aa34b57837e2e5ac874479a 2023-04-06 17:57:04,545 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/B of 5c47f717ab27776f2d38f585a39f2180 into 3a0096183aa34b57837e2e5ac874479a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:04,545 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:04,545 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/B, priority=13, startTime=1680803824107; duration=0sec 2023-04-06 17:57:04,545 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:04,546 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:04,547 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36030 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:04,547 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/C is initiating minor compaction (all files) 2023-04-06 17:57:04,547 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/C in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:04,547 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/2db9d70140534c038d42c3f8dc744717, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/1cfd6c957caf4fa58a37f8142f645151, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/ff35fc0e180c4b408e67c0adb1e287aa] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=35.2 K 2023-04-06 17:57:04,547 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2db9d70140534c038d42c3f8dc744717, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680803820177 2023-04-06 17:57:04,547 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 1cfd6c957caf4fa58a37f8142f645151, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680803820543 2023-04-06 17:57:04,548 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting ff35fc0e180c4b408e67c0adb1e287aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680803822695 2023-04-06 17:57:04,555 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#C#compaction#150 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:04,566 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/276aade37b6d46ba9e3d17baced1a96d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/276aade37b6d46ba9e3d17baced1a96d 2023-04-06 17:57:04,582 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/C of 5c47f717ab27776f2d38f585a39f2180 into 276aade37b6d46ba9e3d17baced1a96d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:04,582 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:04,582 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/C, priority=13, startTime=1680803824107; duration=0sec 2023-04-06 17:57:04,582 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:04,665 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406a1c53537100f4556bdd79ecb441b2e09 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a1c53537100f4556bdd79ecb441b2e09 2023-04-06 17:57:04,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:04,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803884734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:04,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803884740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,742 DEBUG [Thread-951] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:04,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:04,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803884744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,747 DEBUG [Thread-947] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803884748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:04,750 DEBUG [Thread-953] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:04,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:04,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803884816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:05,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803885038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,070 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/bb75e3d003fc4b31af3df7584cf6de6a 2023-04-06 17:57:05,364 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/4bcbae3b8d1f408cb8d3bf37de7267e9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4bcbae3b8d1f408cb8d3bf37de7267e9 2023-04-06 17:57:05,369 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/A of 5c47f717ab27776f2d38f585a39f2180 into 4bcbae3b8d1f408cb8d3bf37de7267e9(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 17:57:05,369 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:05,369 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/A, priority=13, startTime=1680803824107; duration=1sec 2023-04-06 17:57:05,369 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:05,479 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/9d72825cb7304f7d9f83a91917ad894b 2023-04-06 17:57:05,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803885541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,889 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/cd96008bd381462e9aca5cac1dae82f4 2023-04-06 17:57:05,893 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/bb75e3d003fc4b31af3df7584cf6de6a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/bb75e3d003fc4b31af3df7584cf6de6a 2023-04-06 17:57:05,897 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/bb75e3d003fc4b31af3df7584cf6de6a, entries=150, sequenceid=165, filesize=25.3 K 2023-04-06 17:57:05,898 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/9d72825cb7304f7d9f83a91917ad894b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9d72825cb7304f7d9f83a91917ad894b 2023-04-06 17:57:05,902 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9d72825cb7304f7d9f83a91917ad894b, entries=150, sequenceid=165, filesize=11.8 K 2023-04-06 17:57:05,903 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/cd96008bd381462e9aca5cac1dae82f4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cd96008bd381462e9aca5cac1dae82f4 2023-04-06 17:57:05,907 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cd96008bd381462e9aca5cac1dae82f4, entries=150, sequenceid=165, filesize=11.8 K 2023-04-06 17:57:05,908 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5c47f717ab27776f2d38f585a39f2180 in 1666ms, sequenceid=165, compaction requested=false 2023-04-06 17:57:05,908 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:05,908 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:05,908 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:05,908 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:05,908 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,910 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:05,910 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:05,911 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:05,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:05,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:05,913 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:05,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71138efe[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:05,913 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,913 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:05,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:05,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:05,913 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,914 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:05,914 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:05,914 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,914 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:05,914 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:05,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:05,916 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:05,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:05,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:05,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:05,916 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:05,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:05,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:05,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:05,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:05,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:05,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:05,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:05,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:05,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:05,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:05,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:05,923 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:05,923 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:05,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:05,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:05,923 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:05,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:05,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:06,023 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:06,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:06,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:06,026 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:06,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:06,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:06,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:06,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:06,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:06,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:06,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:06,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:06,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:06,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:06,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:06,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:06,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:06,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:06,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:06,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:06,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:06,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:06,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:06,029 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:06,029 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:06,030 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:57:06,030 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:06,030 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:06,030 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:06,030 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:06,030 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:06,030 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:06,041 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406dedd1e1c54484c158141be0b5252e150 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406dedd1e1c54484c158141be0b5252e150 2023-04-06 17:57:06,446 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/0e9e40ba210741e3bbcd9395972a40c4 2023-04-06 17:57:06,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:06,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:06,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:06,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803886582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:06,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:06,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803886685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:06,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:06,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803886821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:06,823 DEBUG [Thread-945] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4125 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:06,857 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/eafade9dd1f841239e57f0a74ebc68b6 2023-04-06 17:57:06,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:06,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57600 deadline: 1680803886886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:07,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803887191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,266 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/4365a22004064165a7a6ab2fbfa456fb 2023-04-06 17:57:07,271 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/0e9e40ba210741e3bbcd9395972a40c4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/0e9e40ba210741e3bbcd9395972a40c4 2023-04-06 17:57:07,275 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/0e9e40ba210741e3bbcd9395972a40c4, entries=150, sequenceid=181, filesize=25.3 K 2023-04-06 17:57:07,276 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/eafade9dd1f841239e57f0a74ebc68b6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/eafade9dd1f841239e57f0a74ebc68b6 2023-04-06 17:57:07,280 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/eafade9dd1f841239e57f0a74ebc68b6, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 17:57:07,281 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/4365a22004064165a7a6ab2fbfa456fb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4365a22004064165a7a6ab2fbfa456fb 2023-04-06 17:57:07,285 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4365a22004064165a7a6ab2fbfa456fb, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 17:57:07,285 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5c47f717ab27776f2d38f585a39f2180 in 1255ms, sequenceid=181, compaction requested=true 2023-04-06 17:57:07,285 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:07,285 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:07,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:07,285 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:07,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:07,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:07,286 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:07,287 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,287 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:07,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:07,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22cebf8e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:07,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:07,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,288 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,289 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:07,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:07,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:07,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:07,289 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:07,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:07,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:07,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:07,291 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:07,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:07,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:07,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:07,291 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:07,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:07,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:07,292 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:07,293 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:07,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:07,296 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:07,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:07,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:07,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:07,296 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:07,296 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:07,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,396 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:07,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:07,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:07,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:07,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:07,401 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:07,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:07,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:07,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:07,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:07,402 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:07,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:07,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:07,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:07,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:07,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:07,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:07,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:07,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:07,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:07,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:07,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:07,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:07,404 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:07,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:07,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:07,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:07,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:07,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803887738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,820 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040602a3d046169440ed92da3790853a6b17 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040602a3d046169440ed92da3790853a6b17 2023-04-06 17:57:07,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:07,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803887842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:07,856 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=200, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/b96efcf3ad71430da0232aabca43e6eb 2023-04-06 17:57:07,935 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/b94050c737664ab79f4aee4d40bbfdcc 2023-04-06 17:57:08,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:08,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803888047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:08,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803888352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,398 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/0f8158a0f392478985950b871fbc3d61 2023-04-06 17:57:08,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/b96efcf3ad71430da0232aabca43e6eb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b96efcf3ad71430da0232aabca43e6eb 2023-04-06 17:57:08,413 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b96efcf3ad71430da0232aabca43e6eb, entries=150, sequenceid=200, filesize=25.3 K 2023-04-06 17:57:08,415 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/b94050c737664ab79f4aee4d40bbfdcc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/b94050c737664ab79f4aee4d40bbfdcc 2023-04-06 17:57:08,434 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/b94050c737664ab79f4aee4d40bbfdcc, entries=150, sequenceid=200, filesize=11.8 K 2023-04-06 17:57:08,435 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/0f8158a0f392478985950b871fbc3d61 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/0f8158a0f392478985950b871fbc3d61 2023-04-06 17:57:08,444 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/0f8158a0f392478985950b871fbc3d61, entries=150, sequenceid=200, filesize=11.8 K 2023-04-06 17:57:08,445 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5c47f717ab27776f2d38f585a39f2180 in 1041ms, sequenceid=200, compaction requested=true 2023-04-06 17:57:08,446 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:08,446 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:08,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,449 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:08,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:08,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38f17fbb[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:08,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:08,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:08,462 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:08,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:08,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:08,467 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:08,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:08,470 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:08,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:08,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:08,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:08,470 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:08,475 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,475 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,475 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:08,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:08,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:08,479 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:08,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:08,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:08,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:08,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:08,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:08,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:08,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:08,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:08,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:08,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:08,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,485 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:08,485 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:08,585 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:08,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:08,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:08,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:08,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:08,601 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:08,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:08,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:08,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:08,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:08,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:08,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:08,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:08,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:08,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:08,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:08,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:08,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:08,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:08,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:08,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:08,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:08,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:08,622 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:08,622 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:08,622 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:57:08,623 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:08,623 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:08,623 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:08,623 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:08,623 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:08,623 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:08,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:08,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:08,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803888909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:09,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803889012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,104 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406804e33983ccd4fb1a23468fb92b864c8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406804e33983ccd4fb1a23468fb92b864c8 2023-04-06 17:57:09,120 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/c9240783d37f45a1b46984cb22425705 2023-04-06 17:57:09,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:09,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803889222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:09,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803889527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,556 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/cfe03e9fb55245608c2b136d5f776c08 2023-04-06 17:57:09,966 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/28352f417eb24bcdbba60d82c3a6fec8 2023-04-06 17:57:09,970 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/c9240783d37f45a1b46984cb22425705 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c9240783d37f45a1b46984cb22425705 2023-04-06 17:57:09,974 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c9240783d37f45a1b46984cb22425705, entries=150, sequenceid=213, filesize=25.3 K 2023-04-06 17:57:09,974 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/cfe03e9fb55245608c2b136d5f776c08 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cfe03e9fb55245608c2b136d5f776c08 2023-04-06 17:57:09,978 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cfe03e9fb55245608c2b136d5f776c08, entries=150, sequenceid=213, filesize=11.8 K 2023-04-06 17:57:09,979 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/28352f417eb24bcdbba60d82c3a6fec8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/28352f417eb24bcdbba60d82c3a6fec8 2023-04-06 17:57:09,982 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/28352f417eb24bcdbba60d82c3a6fec8, entries=150, sequenceid=213, filesize=11.8 K 2023-04-06 17:57:09,983 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5c47f717ab27776f2d38f585a39f2180 in 1361ms, sequenceid=213, compaction requested=true 2023-04-06 17:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:09,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:09,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:09,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:09,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:09,984 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:09,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:09,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:09,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:09,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:09,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:09,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@399b7555[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:09,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:09,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:09,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,000 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,002 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,002 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,005 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,005 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,005 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,005 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:10,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,006 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,005 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,006 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,006 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:10,034 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:57:10,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:10,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:10,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:10,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:10,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:10,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:10,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:10,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803890060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,106 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,114 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,115 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,119 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,119 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@541278e6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,121 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,124 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,124 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,129 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,129 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,129 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:10,130 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,130 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:10,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803890164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,230 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,233 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,235 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,238 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,238 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f78f25[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,239 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,240 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,242 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,242 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,246 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,247 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,247 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,347 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,350 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,351 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,353 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30483447[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,355 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,355 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,355 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,358 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,358 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,361 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:10,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803890368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,375 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,376 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,376 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,450 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406897af3a4a1b2424184fa3d51febc18e1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406897af3a4a1b2424184fa3d51febc18e1 2023-04-06 17:57:10,456 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=233, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/b20a2ac5d4794a7f871a447e6edb6053 2023-04-06 17:57:10,476 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,480 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,481 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,481 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,481 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,482 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,482 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,482 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,482 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,483 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521bdec6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,486 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,489 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,489 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,492 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,494 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,494 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,594 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,597 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,597 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,597 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,599 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,608 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,608 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,608 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,608 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,617 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1509492b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,619 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,619 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,621 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,621 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,621 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,622 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,622 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,622 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,622 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,622 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,624 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,626 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,626 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,625 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:10,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803890671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,726 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,728 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,729 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,731 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,731 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,731 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,731 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab87208[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,733 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,733 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,734 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,734 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,734 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,734 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,736 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,736 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,736 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,737 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,737 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,737 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,737 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,737 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:10,740 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,741 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,841 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,843 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,848 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b238114[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,848 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,849 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,852 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,852 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,853 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,853 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,855 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:10,855 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,855 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:10,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680803890855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,858 DEBUG [Thread-945] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:10,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/cc927a650a3146f1bd29b28850c4befd 2023-04-06 17:57:10,956 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:10,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:10,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:10,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:10,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:10,959 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:10,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:10,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:10,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:10,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:10,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:10,966 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:10,966 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:10,967 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:10,967 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:10,974 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,975 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,975 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,975 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:10,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57429958[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:10,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:10,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,977 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:10,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:10,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:10,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:10,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:10,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,978 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:10,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:10,981 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:10,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:10,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:10,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:10,981 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:10,982 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,982 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:10,983 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:10,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:10,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:10,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:10,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:10,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:10,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:10,987 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:10,987 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:10,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:11,087 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:11,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:11,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:11,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:11,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:11,090 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:11,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:11,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:11,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,090 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,091 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:11,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:11,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:11,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:11,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:11,092 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,092 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:11,092 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:11,092 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:11,092 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:11,093 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,093 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,093 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,093 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:11,094 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:11,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8ddc28a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:11,094 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:11,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:11,096 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,096 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:11,096 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,096 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:11,097 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:11,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:11,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:11,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:11,097 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:11,098 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,098 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,098 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,102 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:11,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,102 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:11,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,102 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:11,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:11,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:11,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803891173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,202 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:11,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:11,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:11,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:11,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:11,205 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:11,205 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,205 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:11,205 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:11,205 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,205 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:11,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:11,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:11,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:11,207 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,207 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:11,207 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:11,207 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:11,208 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,209 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,209 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:11,210 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:11,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619cc9ab[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:11,210 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,211 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,211 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:11,211 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:11,211 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:11,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:11,211 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:11,212 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:11,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:11,214 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:11,214 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:11,214 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:11,214 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:11,214 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:11,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,215 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,218 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:11,218 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:11,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:11,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:11,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:11,218 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:11,218 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:11,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:11,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/e0214068f0ea43eabcecd575d5e53ded 2023-04-06 17:57:11,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/b20a2ac5d4794a7f871a447e6edb6053 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b20a2ac5d4794a7f871a447e6edb6053 2023-04-06 17:57:11,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b20a2ac5d4794a7f871a447e6edb6053, entries=200, sequenceid=233, filesize=32.1 K 2023-04-06 17:57:11,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/cc927a650a3146f1bd29b28850c4befd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cc927a650a3146f1bd29b28850c4befd 2023-04-06 17:57:11,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cc927a650a3146f1bd29b28850c4befd, entries=150, sequenceid=233, filesize=11.8 K 2023-04-06 17:57:11,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/e0214068f0ea43eabcecd575d5e53ded as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e0214068f0ea43eabcecd575d5e53ded 2023-04-06 17:57:11,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e0214068f0ea43eabcecd575d5e53ded, entries=150, sequenceid=233, filesize=11.8 K 2023-04-06 17:57:11,309 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5c47f717ab27776f2d38f585a39f2180 in 1275ms, sequenceid=233, compaction requested=true 2023-04-06 17:57:11,309 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:11,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:11,309 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:57:11,309 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:57:11,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:11,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:11,311 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72490 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:57:11,311 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/B is initiating minor compaction (all files) 2023-04-06 17:57:11,311 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/B in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,311 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/3a0096183aa34b57837e2e5ac874479a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9d72825cb7304f7d9f83a91917ad894b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/eafade9dd1f841239e57f0a74ebc68b6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/b94050c737664ab79f4aee4d40bbfdcc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cfe03e9fb55245608c2b136d5f776c08, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cc927a650a3146f1bd29b28850c4befd] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=70.8 K 2023-04-06 17:57:11,312 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162640 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:57:11,312 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/A is initiating minor compaction (all files) 2023-04-06 17:57:11,312 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/A in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,312 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4bcbae3b8d1f408cb8d3bf37de7267e9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/bb75e3d003fc4b31af3df7584cf6de6a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/0e9e40ba210741e3bbcd9395972a40c4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b96efcf3ad71430da0232aabca43e6eb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c9240783d37f45a1b46984cb22425705, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b20a2ac5d4794a7f871a447e6edb6053] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=158.8 K 2023-04-06 17:57:11,312 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3a0096183aa34b57837e2e5ac874479a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680803822695 2023-04-06 17:57:11,312 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4bcbae3b8d1f408cb8d3bf37de7267e9, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680803822695 2023-04-06 17:57:11,313 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9d72825cb7304f7d9f83a91917ad894b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680803823255 2023-04-06 17:57:11,313 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting bb75e3d003fc4b31af3df7584cf6de6a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680803823255 2023-04-06 17:57:11,313 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting eafade9dd1f841239e57f0a74ebc68b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803824404 2023-04-06 17:57:11,313 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 0e9e40ba210741e3bbcd9395972a40c4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803824404 2023-04-06 17:57:11,313 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b94050c737664ab79f4aee4d40bbfdcc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680803826573 2023-04-06 17:57:11,314 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b96efcf3ad71430da0232aabca43e6eb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680803826573 2023-04-06 17:57:11,314 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cfe03e9fb55245608c2b136d5f776c08, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680803827723 2023-04-06 17:57:11,314 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c9240783d37f45a1b46984cb22425705, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680803827723 2023-04-06 17:57:11,314 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b20a2ac5d4794a7f871a447e6edb6053, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803828902 2023-04-06 17:57:11,314 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cc927a650a3146f1bd29b28850c4befd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803828902 2023-04-06 17:57:11,318 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:11,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:11,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:11,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:11,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:11,324 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:11,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:11,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:11,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:11,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:11,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:11,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:11,328 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:11,328 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,328 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:11,328 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:11,328 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:11,328 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,329 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:11,329 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:57:11,329 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:11,329 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:11,329 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:11,329 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:11,329 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:11,329 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:11,351 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#B#compaction#166 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:11,352 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#A#compaction#165 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:11,371 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068b46d8dd4dd146ed9b2bb2a1a814b054 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068b46d8dd4dd146ed9b2bb2a1a814b054 2023-04-06 17:57:11,383 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/093864af92634efead19ed61963de794 2023-04-06 17:57:11,398 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/dec698bc3fbc4448a5802d18e1591b6c 2023-04-06 17:57:11,764 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/5933947a45c546579582e95decaeb5fd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/5933947a45c546579582e95decaeb5fd 2023-04-06 17:57:11,769 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/B of 5c47f717ab27776f2d38f585a39f2180 into 5933947a45c546579582e95decaeb5fd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:11,769 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:11,769 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/B, priority=10, startTime=1680803831309; duration=0sec 2023-04-06 17:57:11,769 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:11,769 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:57:11,771 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72490 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:57:11,771 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/C is initiating minor compaction (all files) 2023-04-06 17:57:11,771 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/C in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,771 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/276aade37b6d46ba9e3d17baced1a96d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cd96008bd381462e9aca5cac1dae82f4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4365a22004064165a7a6ab2fbfa456fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/0f8158a0f392478985950b871fbc3d61, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/28352f417eb24bcdbba60d82c3a6fec8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e0214068f0ea43eabcecd575d5e53ded] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=70.8 K 2023-04-06 17:57:11,771 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 276aade37b6d46ba9e3d17baced1a96d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680803822695 2023-04-06 17:57:11,772 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cd96008bd381462e9aca5cac1dae82f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680803823255 2023-04-06 17:57:11,772 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4365a22004064165a7a6ab2fbfa456fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803824404 2023-04-06 17:57:11,772 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0f8158a0f392478985950b871fbc3d61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680803826573 2023-04-06 17:57:11,772 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 28352f417eb24bcdbba60d82c3a6fec8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680803827723 2023-04-06 17:57:11,773 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e0214068f0ea43eabcecd575d5e53ded, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803828902 2023-04-06 17:57:11,787 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#C#compaction#170 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:11,801 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/02cbeed2a1ab474484e2e636359cafae as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02cbeed2a1ab474484e2e636359cafae 2023-04-06 17:57:11,802 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/c4ac5bdcc8f44c27983c5f30ccdc7fe7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/c4ac5bdcc8f44c27983c5f30ccdc7fe7 2023-04-06 17:57:11,808 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/e42bcdcbe56746b0ba68b8038304eaac 2023-04-06 17:57:11,813 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/A of 5c47f717ab27776f2d38f585a39f2180 into 02cbeed2a1ab474484e2e636359cafae(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:11,813 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:11,813 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/A, priority=10, startTime=1680803831309; duration=0sec 2023-04-06 17:57:11,813 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:11,815 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/093864af92634efead19ed61963de794 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/093864af92634efead19ed61963de794 2023-04-06 17:57:11,818 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/C of 5c47f717ab27776f2d38f585a39f2180 into c4ac5bdcc8f44c27983c5f30ccdc7fe7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:11,818 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:11,818 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/C, priority=10, startTime=1680803831309; duration=0sec 2023-04-06 17:57:11,818 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:11,820 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/093864af92634efead19ed61963de794, entries=150, sequenceid=245, filesize=25.3 K 2023-04-06 17:57:11,821 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/dec698bc3fbc4448a5802d18e1591b6c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dec698bc3fbc4448a5802d18e1591b6c 2023-04-06 17:57:11,825 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dec698bc3fbc4448a5802d18e1591b6c, entries=150, sequenceid=245, filesize=11.8 K 2023-04-06 17:57:11,826 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/e42bcdcbe56746b0ba68b8038304eaac as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e42bcdcbe56746b0ba68b8038304eaac 2023-04-06 17:57:11,830 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e42bcdcbe56746b0ba68b8038304eaac, entries=150, sequenceid=245, filesize=11.8 K 2023-04-06 17:57:11,830 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 5c47f717ab27776f2d38f585a39f2180 in 501ms, sequenceid=245, compaction requested=false 2023-04-06 17:57:11,831 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:11,831 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:11,834 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,834 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:11,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2670dfad[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:11,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:11,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:11,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:11,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:11,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:11,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:11,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:11,844 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:11,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:11,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:11,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:11,844 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:11,848 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,848 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:11,848 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:11,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:11,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:11,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,852 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:11,853 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:11,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:11,953 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:11,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:11,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:11,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:11,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:11,956 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:11,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:11,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:11,957 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,957 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:11,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:11,957 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:11,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:11,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:11,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:11,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:11,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:11,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:11,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:11,960 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,960 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:11,960 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:11,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:11,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,974 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,974 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:11,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bb0a62[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:11,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:11,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,983 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:11,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:11,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:11,983 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:11,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:11,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:11,990 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:11,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:11,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:11,992 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:11,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:11,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:11,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:11,992 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:11,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,000 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,000 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:12,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:12,001 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:12,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:12,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:12,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:12,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:12,007 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:12,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:12,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,007 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:12,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:12,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:12,008 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,008 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,008 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:12,107 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:12,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:12,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:12,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:12,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:12,114 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:12,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:12,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:12,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:12,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,115 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:12,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:12,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:12,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:12,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:12,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:12,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:12,117 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:12,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:12,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:12,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:12,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,119 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3145dcd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:12,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:12,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:12,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:12,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:12,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:12,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:12,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:12,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:12,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:12,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:12,125 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:12,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:12,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:12,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:12,126 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:12,127 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,127 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,127 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:12,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:12,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,129 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:12,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:12,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,131 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:12,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:12,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:12,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:12,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:12,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,131 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:12,131 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:12,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:12,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:12,231 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:12,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:12,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:12,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:12,235 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:12,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:12,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:12,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:12,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:12,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:12,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:12,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:12,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:12,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:12,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:12,243 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:12,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:12,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:12,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:12,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803832340, max timestamp=1680803832354Num uniques 859; 2023-04-06 17:57:12,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803832340, max timestamp=1680803832354Num uniques 859; 2023-04-06 17:57:12,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803832340, max timestamp=1680803832354Num uniques 859; 2023-04-06 17:57:12,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:12,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:12,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:12,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680803832378, max timestamp=1680803832410Num uniques 1018; 2023-04-06 17:57:12,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 17:57:12,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680803832378, max timestamp=1680803832410Num uniques 1018; 2023-04-06 17:57:12,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680803832378, max timestamp=1680803832410Num uniques 1018; 2023-04-06 17:57:12,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 17:57:12,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 17:57:12,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680803832421, max timestamp=1680803832452Num uniques 1027; 2023-04-06 17:57:12,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 17:57:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680803832421, max timestamp=1680803832452Num uniques 1027; 2023-04-06 17:57:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 17:57:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680803832421, max timestamp=1680803832452Num uniques 1027; 2023-04-06 17:57:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 17:57:12,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680803832479, max timestamp=1680803832488Num uniques 1086; 2023-04-06 17:57:12,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-06 17:57:12,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680803832479, max timestamp=1680803832488Num uniques 1086; 2023-04-06 17:57:12,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-06 17:57:12,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680803832479, max timestamp=1680803832488Num uniques 1086; 2023-04-06 17:57:12,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-06 17:57:12,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680803832519, max timestamp=1680803832528Num uniques 1095; 2023-04-06 17:57:12,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-06 17:57:12,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680803832519, max timestamp=1680803832528Num uniques 1095; 2023-04-06 17:57:12,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-06 17:57:12,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680803832519, max timestamp=1680803832528Num uniques 1095; 2023-04-06 17:57:12,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-06 17:57:12,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680803832537, max timestamp=1680803832584Num uniques 1104; 2023-04-06 17:57:12,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680803832537, max timestamp=1680803832584Num uniques 1104; 2023-04-06 17:57:12,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 17:57:12,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 17:57:12,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680803832537, max timestamp=1680803832584Num uniques 1104; 2023-04-06 17:57:12,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 17:57:12,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680803832603, max timestamp=1680803832626Num uniques 1063; 2023-04-06 17:57:12,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 17:57:12,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680803832603, max timestamp=1680803832626Num uniques 1063; 2023-04-06 17:57:12,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 17:57:12,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680803832603, max timestamp=1680803832626Num uniques 1063; 2023-04-06 17:57:12,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 17:57:12,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680803832661, max timestamp=1680803832670Num uniques 1072; 2023-04-06 17:57:12,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-06 17:57:12,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680803832661, max timestamp=1680803832670Num uniques 1072; 2023-04-06 17:57:12,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-06 17:57:12,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680803832661, max timestamp=1680803832670Num uniques 1072; 2023-04-06 17:57:12,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-06 17:57:12,686 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b8010c06e2f54019bddbef1b499ddc31 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b8010c06e2f54019bddbef1b499ddc31 2023-04-06 17:57:12,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680803832699, max timestamp=1680803832715Num uniques 1031; 2023-04-06 17:57:12,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-06 17:57:12,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680803832699, max timestamp=1680803832715Num uniques 1031; 2023-04-06 17:57:12,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-06 17:57:12,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680803832699, max timestamp=1680803832715Num uniques 1031; 2023-04-06 17:57:12,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-06 17:57:12,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680803832743, max timestamp=1680803832762Num uniques 1090; 2023-04-06 17:57:12,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-06 17:57:12,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680803832743, max timestamp=1680803832762Num uniques 1090; 2023-04-06 17:57:12,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-06 17:57:12,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680803832743, max timestamp=1680803832762Num uniques 1090; 2023-04-06 17:57:12,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-06 17:57:12,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680803832783, max timestamp=1680803832795Num uniques 1099; 2023-04-06 17:57:12,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680803832783, max timestamp=1680803832795Num uniques 1099; 2023-04-06 17:57:12,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-06 17:57:12,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680803832783, max timestamp=1680803832795Num uniques 1099; 2023-04-06 17:57:12,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-06 17:57:12,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-06 17:57:12,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680803832831, max timestamp=1680803832841Num uniques 1108; 2023-04-06 17:57:12,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680803832831, max timestamp=1680803832841Num uniques 1108; 2023-04-06 17:57:12,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-06 17:57:12,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680803832831, max timestamp=1680803832841Num uniques 1108; 2023-04-06 17:57:12,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-06 17:57:12,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-06 17:57:12,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680803832857, max timestamp=1680803832876Num uniques 1067; 2023-04-06 17:57:12,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680803832857, max timestamp=1680803832876Num uniques 1067; 2023-04-06 17:57:12,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680803832857, max timestamp=1680803832876Num uniques 1067; 2023-04-06 17:57:12,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-06 17:57:12,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-06 17:57:12,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-06 17:57:12,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680803832900, max timestamp=1680803832910Num uniques 1126; 2023-04-06 17:57:12,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-06 17:57:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680803832900, max timestamp=1680803832910Num uniques 1126; 2023-04-06 17:57:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-06 17:57:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680803832900, max timestamp=1680803832910Num uniques 1126; 2023-04-06 17:57:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-06 17:57:12,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680803832921, max timestamp=1680803832951Num uniques 1085; 2023-04-06 17:57:12,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-06 17:57:12,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680803832921, max timestamp=1680803832951Num uniques 1085; 2023-04-06 17:57:12,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680803832921, max timestamp=1680803832951Num uniques 1085; 2023-04-06 17:57:12,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-06 17:57:12,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-06 17:57:12,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680803832971, max timestamp=1680803832981Num uniques 1094; 2023-04-06 17:57:12,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-06 17:57:12,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680803832971, max timestamp=1680803832981Num uniques 1094; 2023-04-06 17:57:12,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680803832971, max timestamp=1680803832981Num uniques 1094; 2023-04-06 17:57:12,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-06 17:57:12,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-06 17:57:13,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680803833006, max timestamp=1680803833035Num uniques 1053; 2023-04-06 17:57:13,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-06 17:57:13,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680803833006, max timestamp=1680803833035Num uniques 1053; 2023-04-06 17:57:13,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680803833006, max timestamp=1680803833035Num uniques 1053; 2023-04-06 17:57:13,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-06 17:57:13,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-06 17:57:13,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680803833075, max timestamp=1680803833085Num uniques 1012; 2023-04-06 17:57:13,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680803833075, max timestamp=1680803833085Num uniques 1012; 2023-04-06 17:57:13,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-06 17:57:13,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-06 17:57:13,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680803833075, max timestamp=1680803833085Num uniques 1012; 2023-04-06 17:57:13,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-06 17:57:13,093 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/df81ab24bab24d7880cb613bc367af7a 2023-04-06 17:57:13,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680803833113, max timestamp=1680803833122Num uniques 1021; 2023-04-06 17:57:13,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-06 17:57:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680803833113, max timestamp=1680803833122Num uniques 1021; 2023-04-06 17:57:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-06 17:57:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680803833113, max timestamp=1680803833122Num uniques 1021; 2023-04-06 17:57:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-06 17:57:13,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680803833152, max timestamp=1680803833177Num uniques 1030; 2023-04-06 17:57:13,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-06 17:57:13,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680803833152, max timestamp=1680803833177Num uniques 1030; 2023-04-06 17:57:13,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-06 17:57:13,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680803833152, max timestamp=1680803833177Num uniques 1030; 2023-04-06 17:57:13,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-06 17:57:13,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680803833206, max timestamp=1680803833225Num uniques 1139; 2023-04-06 17:57:13,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680803833206, max timestamp=1680803833225Num uniques 1139; 2023-04-06 17:57:13,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680803833206, max timestamp=1680803833225Num uniques 1139; 2023-04-06 17:57:13,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-06 17:57:13,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-06 17:57:13,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-06 17:57:13,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680803833247, max timestamp=1680803833261Num uniques 1098; 2023-04-06 17:57:13,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-06 17:57:13,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680803833247, max timestamp=1680803833261Num uniques 1098; 2023-04-06 17:57:13,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-06 17:57:13,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680803833247, max timestamp=1680803833261Num uniques 1098; 2023-04-06 17:57:13,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-06 17:57:13,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680803833280, max timestamp=1680803833295Num uniques 1107; 2023-04-06 17:57:13,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-06 17:57:13,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680803833280, max timestamp=1680803833295Num uniques 1107; 2023-04-06 17:57:13,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-06 17:57:13,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680803833280, max timestamp=1680803833295Num uniques 1107; 2023-04-06 17:57:13,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-06 17:57:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680803833325, max timestamp=1680803833347Num uniques 1016; 2023-04-06 17:57:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-04-06 17:57:13,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680803833325, max timestamp=1680803833347Num uniques 1016; 2023-04-06 17:57:13,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680803833325, max timestamp=1680803833347Num uniques 1016; 2023-04-06 17:57:13,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-04-06 17:57:13,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-04-06 17:57:13,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680803833368, max timestamp=1680803833377Num uniques 1075; 2023-04-06 17:57:13,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-06 17:57:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680803833368, max timestamp=1680803833377Num uniques 1075; 2023-04-06 17:57:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-06 17:57:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680803833368, max timestamp=1680803833377Num uniques 1075; 2023-04-06 17:57:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-06 17:57:13,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680803833401, max timestamp=1680803833411Num uniques 1084; 2023-04-06 17:57:13,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680803833401, max timestamp=1680803833411Num uniques 1084; 2023-04-06 17:57:13,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-06 17:57:13,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-06 17:57:13,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680803833401, max timestamp=1680803833411Num uniques 1084; 2023-04-06 17:57:13,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-06 17:57:13,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1680803833439, max timestamp=1680803833454Num uniques 1093; 2023-04-06 17:57:13,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-06 17:57:13,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1680803833439, max timestamp=1680803833454Num uniques 1093; 2023-04-06 17:57:13,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1680803833439, max timestamp=1680803833454Num uniques 1093; 2023-04-06 17:57:13,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-06 17:57:13,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-06 17:57:13,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680803833468, max timestamp=1680803833489Num uniques 1102; 2023-04-06 17:57:13,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680803833468, max timestamp=1680803833489Num uniques 1102; 2023-04-06 17:57:13,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=11540, suffix heap size=142576,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=252 2023-04-06 17:57:13,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=11540, suffix heap size=142576,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=252 2023-04-06 17:57:13,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680803833468, max timestamp=1680803833489Num uniques 1102; 2023-04-06 17:57:13,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=11540, suffix heap size=142576,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=252 2023-04-06 17:57:13,503 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/8e8c87eb46a94e33a564323bf0e00126 2023-04-06 17:57:13,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680803833545, max timestamp=1680803833561Num uniques 1111; 2023-04-06 17:57:13,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=11946, suffix heap size=143512,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=261 2023-04-06 17:57:13,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680803833545, max timestamp=1680803833561Num uniques 1111; 2023-04-06 17:57:13,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=11946, suffix heap size=143512,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=261 2023-04-06 17:57:13,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680803833545, max timestamp=1680803833561Num uniques 1111; 2023-04-06 17:57:13,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=11946, suffix heap size=143512,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=261 2023-04-06 17:57:13,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1680803833576, max timestamp=1680803833600Num uniques 1120; 2023-04-06 17:57:13,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=10070, suffix heap size=144448,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=220 2023-04-06 17:57:13,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1680803833576, max timestamp=1680803833600Num uniques 1120; 2023-04-06 17:57:13,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=10070, suffix heap size=144448,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=220 2023-04-06 17:57:13,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1680803833576, max timestamp=1680803833600Num uniques 1120; 2023-04-06 17:57:13,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=10070, suffix heap size=144448,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=220 2023-04-06 17:57:13,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1680803833607, max timestamp=1680803833639Num uniques 1079; 2023-04-06 17:57:13,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1680803833607, max timestamp=1680803833639Num uniques 1079; 2023-04-06 17:57:13,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-06 17:57:13,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-06 17:57:13,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1680803833607, max timestamp=1680803833639Num uniques 1079; 2023-04-06 17:57:13,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-06 17:57:13,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680803833661, max timestamp=1680803833686Num uniques 1038; 2023-04-06 17:57:13,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680803833661, max timestamp=1680803833686Num uniques 1038; 2023-04-06 17:57:13,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-06 17:57:13,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-06 17:57:13,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680803833661, max timestamp=1680803833686Num uniques 1038; 2023-04-06 17:57:13,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-06 17:57:13,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1680803833701, max timestamp=1680803833732Num uniques 1097; 2023-04-06 17:57:13,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-06 17:57:13,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1680803833701, max timestamp=1680803833732Num uniques 1097; 2023-04-06 17:57:13,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-06 17:57:13,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1680803833701, max timestamp=1680803833732Num uniques 1097; 2023-04-06 17:57:13,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-06 17:57:13,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1680803833765, max timestamp=1680803833772Num uniques 1106; 2023-04-06 17:57:13,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-06 17:57:13,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1680803833765, max timestamp=1680803833772Num uniques 1106; 2023-04-06 17:57:13,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-06 17:57:13,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1680803833765, max timestamp=1680803833772Num uniques 1106; 2023-04-06 17:57:13,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-06 17:57:13,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1680803833797, max timestamp=1680803833818Num uniques 1015; 2023-04-06 17:57:13,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=7550, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-06 17:57:13,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1680803833797, max timestamp=1680803833818Num uniques 1015; 2023-04-06 17:57:13,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=7550, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-06 17:57:13,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1680803833797, max timestamp=1680803833818Num uniques 1015; 2023-04-06 17:57:13,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=7550, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-06 17:57:13,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680803833839, max timestamp=1680803833851Num uniques 1024; 2023-04-06 17:57:13,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680803833839, max timestamp=1680803833851Num uniques 1024; 2023-04-06 17:57:13,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680803833839, max timestamp=1680803833851Num uniques 1024; 2023-04-06 17:57:13,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-06 17:57:13,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-06 17:57:13,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-06 17:57:13,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=283, cellSize=12958, totalHeapSize=29672, min timestamp=1680803833850, max timestamp=1680803833893Num uniques 1083; 2023-04-06 17:57:13,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=283, cellSize=12958, totalHeapSize=29672, min timestamp=1680803833850, max timestamp=1680803833893Num uniques 1083; 2023-04-06 17:57:13,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=283, cellSize=12958, totalHeapSize=29672, min timestamp=1680803833850, max timestamp=1680803833893Num uniques 1083; 2023-04-06 17:57:13,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=12958, suffix heap size=140600,new segment heap size=29672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=283 2023-04-06 17:57:13,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=12958, suffix heap size=140600,new segment heap size=29672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=283 2023-04-06 17:57:13,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=12958, suffix heap size=140600,new segment heap size=29672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=283 2023-04-06 17:57:13,913 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/17def39147bc4681a82374a0d804f42c 2023-04-06 17:57:13,917 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/df81ab24bab24d7880cb613bc367af7a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/df81ab24bab24d7880cb613bc367af7a 2023-04-06 17:57:13,927 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/df81ab24bab24d7880cb613bc367af7a, entries=150, sequenceid=256, filesize=25.3 K 2023-04-06 17:57:13,928 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/8e8c87eb46a94e33a564323bf0e00126 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/8e8c87eb46a94e33a564323bf0e00126 2023-04-06 17:57:13,936 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/8e8c87eb46a94e33a564323bf0e00126, entries=100, sequenceid=256, filesize=9.4 K 2023-04-06 17:57:13,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=342, cellSize=15662, totalHeapSize=35808, min timestamp=1680803833914, max timestamp=1680803833935Num uniques 1142; 2023-04-06 17:57:13,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=342, cellSize=15662, totalHeapSize=35808, min timestamp=1680803833914, max timestamp=1680803833935Num uniques 1142; 2023-04-06 17:57:13,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52302, new segment data size=15662, suffix heap size=146736,new segment heap size=35808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1142, new segment cells count=342 2023-04-06 17:57:13,940 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/17def39147bc4681a82374a0d804f42c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/17def39147bc4681a82374a0d804f42c 2023-04-06 17:57:13,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1680803833917, max timestamp=1680803833935Num uniques 1142; 2023-04-06 17:57:13,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52302, new segment data size=8792, suffix heap size=146736,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1142, new segment cells count=192 2023-04-06 17:57:13,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52302, new segment data size=15662, suffix heap size=146736,new segment heap size=35808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1142, new segment cells count=342 2023-04-06 17:57:13,946 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/17def39147bc4681a82374a0d804f42c, entries=100, sequenceid=256, filesize=9.4 K 2023-04-06 17:57:13,950 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=46.96 KB/48090 for 5c47f717ab27776f2d38f585a39f2180 in 1707ms, sequenceid=256, compaction requested=true 2023-04-06 17:57:13,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:13,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:13,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:13,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:13,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:13,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:13,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:13,951 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=53.67 KB heapSize=133.45 KB 2023-04-06 17:57:13,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:13,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:13,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:13,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:13,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:13,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:13,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:13,954 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:13,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:13,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c21d95[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:13,957 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:13,957 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,958 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:13,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:13,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:13,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:13,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:13,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:13,966 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:13,966 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:13,966 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:13,966 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:13,966 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:13,974 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:13,974 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:13,975 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:13,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:13,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:13,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:13,977 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:13,980 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068afcc0103078448c9894bef31fd8477a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068afcc0103078448c9894bef31fd8477a 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:13,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:13,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:13,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:13,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:13,984 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:13,985 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:13,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:13,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:13,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:14,004 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=914, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/02fe5a72d7404ecaaed1bec8911c7d70 2023-04-06 17:57:14,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=914 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/0c18d8cf32e34546b3240e26666a8934 2023-04-06 17:57:14,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=914 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/f8829652842a4a8e9eda9af0895a7958 2023-04-06 17:57:14,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/02fe5a72d7404ecaaed1bec8911c7d70 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02fe5a72d7404ecaaed1bec8911c7d70 2023-04-06 17:57:14,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803834011, max timestamp=1680803834036Num uniques 859; 2023-04-06 17:57:14,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:14,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803834011, max timestamp=1680803834036Num uniques 859; 2023-04-06 17:57:14,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:14,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803834011, max timestamp=1680803834036Num uniques 859; 2023-04-06 17:57:14,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:14,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02fe5a72d7404ecaaed1bec8911c7d70, entries=200, sequenceid=914, filesize=32.3 K 2023-04-06 17:57:14,041 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/0c18d8cf32e34546b3240e26666a8934 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/0c18d8cf32e34546b3240e26666a8934 2023-04-06 17:57:14,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/0c18d8cf32e34546b3240e26666a8934, entries=150, sequenceid=914, filesize=11.9 K 2023-04-06 17:57:14,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/f8829652842a4a8e9eda9af0895a7958 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/f8829652842a4a8e9eda9af0895a7958 2023-04-06 17:57:14,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/f8829652842a4a8e9eda9af0895a7958, entries=150, sequenceid=914, filesize=11.9 K 2023-04-06 17:57:14,050 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~133.41 KB/136608, currentSize=60.38 KB/61830 for 5c47f717ab27776f2d38f585a39f2180 in 98ms, sequenceid=914, compaction requested=true 2023-04-06 17:57:14,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:14,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:14,050 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:14,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:14,050 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:14,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:14,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:14,054 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=67.09 KB heapSize=181.08 KB 2023-04-06 17:57:14,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:14,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:14,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:14,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:14,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:14,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:14,063 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46378 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:14,063 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111248 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:14,063 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/B is initiating minor compaction (all files) 2023-04-06 17:57:14,063 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/A is initiating minor compaction (all files) 2023-04-06 17:57:14,063 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/B in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,063 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/A in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,064 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02cbeed2a1ab474484e2e636359cafae, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/093864af92634efead19ed61963de794, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/df81ab24bab24d7880cb613bc367af7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02fe5a72d7404ecaaed1bec8911c7d70] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=108.6 K 2023-04-06 17:57:14,064 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/5933947a45c546579582e95decaeb5fd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dec698bc3fbc4448a5802d18e1591b6c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/8e8c87eb46a94e33a564323bf0e00126, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/0c18d8cf32e34546b3240e26666a8934] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=45.3 K 2023-04-06 17:57:14,064 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 02cbeed2a1ab474484e2e636359cafae, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803828902 2023-04-06 17:57:14,064 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5933947a45c546579582e95decaeb5fd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803828902 2023-04-06 17:57:14,064 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 093864af92634efead19ed61963de794, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680803830045 2023-04-06 17:57:14,064 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting dec698bc3fbc4448a5802d18e1591b6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680803830045 2023-04-06 17:57:14,065 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting df81ab24bab24d7880cb613bc367af7a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680803832202 2023-04-06 17:57:14,065 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 02fe5a72d7404ecaaed1bec8911c7d70, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1680803833923 2023-04-06 17:57:14,065 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 8e8c87eb46a94e33a564323bf0e00126, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680803832226 2023-04-06 17:57:14,065 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0c18d8cf32e34546b3240e26666a8934, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1680803833935 2023-04-06 17:57:14,079 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#B#compaction#177 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:14,085 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:14,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:14,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:14,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:14,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:14,105 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:14,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:14,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:14,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,107 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:14,115 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/fc94cacc99e54b3c9b4af7bf8309b11d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/fc94cacc99e54b3c9b4af7bf8309b11d 2023-04-06 17:57:14,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:14,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:14,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:14,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:14,121 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/B of 5c47f717ab27776f2d38f585a39f2180 into fc94cacc99e54b3c9b4af7bf8309b11d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:14,121 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:14,121 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/B, priority=12, startTime=1680803834050; duration=0sec 2023-04-06 17:57:14,121 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:14,121 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:14,122 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46378 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:14,122 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/C is initiating minor compaction (all files) 2023-04-06 17:57:14,123 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/C in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,123 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/c4ac5bdcc8f44c27983c5f30ccdc7fe7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e42bcdcbe56746b0ba68b8038304eaac, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/17def39147bc4681a82374a0d804f42c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/f8829652842a4a8e9eda9af0895a7958] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=45.3 K 2023-04-06 17:57:14,123 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c4ac5bdcc8f44c27983c5f30ccdc7fe7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803828902 2023-04-06 17:57:14,123 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e42bcdcbe56746b0ba68b8038304eaac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680803830045 2023-04-06 17:57:14,124 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 17def39147bc4681a82374a0d804f42c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680803832226 2023-04-06 17:57:14,124 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f8829652842a4a8e9eda9af0895a7958, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1680803833935 2023-04-06 17:57:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:14,126 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,126 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:14,126 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:14,126 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 890 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803894125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,136 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#A#compaction#178 average throughput is 0.39 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:14,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:14,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:14,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:14,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24ef2717[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:14,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:14,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:14,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:14,145 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:14,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:14,146 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:14,146 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,146 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:14,147 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#C#compaction#180 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:14,148 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:14,149 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:14,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:14,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:14,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:14,150 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:14,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,151 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,151 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,151 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:14,155 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:14,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:14,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,155 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:14,155 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:14,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,155 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,155 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,159 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406719f2470084345c89857f6eb36dcb90a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406719f2470084345c89857f6eb36dcb90a 2023-04-06 17:57:14,176 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=940, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/f44d40ff9ecc4617b1c88a528467fb5a 2023-04-06 17:57:14,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 892 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803894231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,255 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:14,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:14,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:14,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:14,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:14,258 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:14,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:14,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:14,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,259 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,259 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,259 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,259 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:14,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:14,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:14,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:14,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:14,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:14,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:14,260 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,261 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:14,261 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:14,261 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:14,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:14,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:14,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:14,262 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,262 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,262 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,262 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:14,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:14,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19bac1f2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:14,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,264 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:14,264 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:14,264 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:14,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:14,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,264 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:14,265 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,265 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:14,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:14,267 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:14,267 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:14,267 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:14,267 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:14,267 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:14,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,269 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,270 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,270 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,270 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,270 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,270 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,270 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:14,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:14,270 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:14,270 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,271 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,271 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:14,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,271 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:14,271 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,371 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:14,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:14,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:14,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:14,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:14,374 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:14,374 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,374 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:14,374 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:14,374 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,374 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:14,378 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:14,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:14,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:14,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:14,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:14,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:14,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:14,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:14,379 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,379 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:14,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:14,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:14,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:14,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:14,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:14,381 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,381 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,381 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:14,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e10d3d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:14,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:14,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:14,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:14,383 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:14,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:14,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:14,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:14,386 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:14,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:14,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:14,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:14,386 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:14,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,387 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:14,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:14,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:14,391 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:14,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,391 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:14,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 894 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803894435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,491 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:14,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:14,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:14,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:14,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:14,494 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:14,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:14,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:14,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:14,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:14,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:14,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:14,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:14,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:14,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:14,496 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,497 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:14,497 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:14,497 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:14,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:14,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:14,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:14,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:14,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cdb8101[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:14,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:14,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,499 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:14,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:14,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:14,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:14,500 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:14,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:14,502 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:14,502 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:14,502 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:14,502 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:14,502 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:14,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,503 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,504 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,505 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,505 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,505 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,505 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:14,505 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:14,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:14,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,505 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,506 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,506 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,506 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:14,506 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:14,506 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,572 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/615820e6e6eb4750a1f967408a2f67c9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/615820e6e6eb4750a1f967408a2f67c9 2023-04-06 17:57:14,578 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/C of 5c47f717ab27776f2d38f585a39f2180 into 615820e6e6eb4750a1f967408a2f67c9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:14,579 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:14,579 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/C, priority=12, startTime=1680803834050; duration=0sec 2023-04-06 17:57:14,579 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:14,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=940 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/52f45a5b42cb45ef9e824f9ecc959650 2023-04-06 17:57:14,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=940 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/964520796ad2487c91bb25084f88caa7 2023-04-06 17:57:14,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/f44d40ff9ecc4617b1c88a528467fb5a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f44d40ff9ecc4617b1c88a528467fb5a 2023-04-06 17:57:14,606 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:14,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:14,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:14,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:14,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:14,609 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:14,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:14,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:14,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f44d40ff9ecc4617b1c88a528467fb5a, entries=200, sequenceid=940, filesize=32.3 K 2023-04-06 17:57:14,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:14,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/52f45a5b42cb45ef9e824f9ecc959650 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/52f45a5b42cb45ef9e824f9ecc959650 2023-04-06 17:57:14,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/52f45a5b42cb45ef9e824f9ecc959650, entries=150, sequenceid=940, filesize=11.9 K 2023-04-06 17:57:14,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/964520796ad2487c91bb25084f88caa7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/964520796ad2487c91bb25084f88caa7 2023-04-06 17:57:14,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:14,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:14,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:14,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:14,627 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:14,627 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,627 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:14,627 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:14,627 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:14,627 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,627 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:14,627 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:14,627 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,627 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:14,627 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:14,629 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,629 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,629 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:14,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/964520796ad2487c91bb25084f88caa7, entries=150, sequenceid=940, filesize=11.9 K 2023-04-06 17:57:14,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~181.03 KB/185376, currentSize=114.05 KB/116790 for 5c47f717ab27776f2d38f585a39f2180 in 576ms, sequenceid=940, compaction requested=false 2023-04-06 17:57:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:14,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d1dc12[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:14,634 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:14,634 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,641 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,641 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:14,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:14,642 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:14,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:14,645 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:14,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:14,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:14,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:14,646 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:14,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,648 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:14,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,652 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:14,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:14,652 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,652 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:14,652 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:14,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:14,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-06 17:57:14,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:14,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:14,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:14,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:14,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:14,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:14,752 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:14,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:14,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803894759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803894760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 903 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803894760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:14,762 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:14,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:14,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:14,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,764 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:14,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:14,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:14,765 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:14,765 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,765 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:14,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:14,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:14,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5671a0f8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,777 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068d8df3558f074214bc367d28ef657fc9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068d8df3558f074214bc367d28ef657fc9 2023-04-06 17:57:14,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:14,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:14,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:14,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:14,783 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=963, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/767b82c0dcce469c87c44785492912fb 2023-04-06 17:57:14,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,783 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:14,783 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,783 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:14,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:14,788 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:14,788 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:14,788 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:14,788 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:14,788 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:14,791 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,791 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,791 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,793 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=963 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/e50f5efba45d4e3fbc66d1e781574b35 2023-04-06 17:57:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,795 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:14,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:14,796 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,795 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,796 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:14,795 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,796 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,796 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,796 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,796 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,796 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,796 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,796 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:14,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=963 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/244538aa58c74bf39aa555c7f1d5a942 2023-04-06 17:57:14,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/767b82c0dcce469c87c44785492912fb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/767b82c0dcce469c87c44785492912fb 2023-04-06 17:57:14,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/767b82c0dcce469c87c44785492912fb, entries=200, sequenceid=963, filesize=32.3 K 2023-04-06 17:57:14,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/e50f5efba45d4e3fbc66d1e781574b35 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/e50f5efba45d4e3fbc66d1e781574b35 2023-04-06 17:57:14,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/e50f5efba45d4e3fbc66d1e781574b35, entries=150, sequenceid=963, filesize=11.9 K 2023-04-06 17:57:14,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/244538aa58c74bf39aa555c7f1d5a942 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/244538aa58c74bf39aa555c7f1d5a942 2023-04-06 17:57:14,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/244538aa58c74bf39aa555c7f1d5a942, entries=150, sequenceid=963, filesize=11.9 K 2023-04-06 17:57:14,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 5c47f717ab27776f2d38f585a39f2180 in 84ms, sequenceid=963, compaction requested=true 2023-04-06 17:57:14,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:14,825 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-06 17:57:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:57:14,826 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:14,826 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:14,826 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. because compaction request was cancelled 2023-04-06 17:57:14,826 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:14,827 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:14,827 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/B is initiating minor compaction (all files) 2023-04-06 17:57:14,827 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/B in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,827 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/fc94cacc99e54b3c9b4af7bf8309b11d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/52f45a5b42cb45ef9e824f9ecc959650, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/e50f5efba45d4e3fbc66d1e781574b35] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=36.3 K 2023-04-06 17:57:14,828 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting fc94cacc99e54b3c9b4af7bf8309b11d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1680803833935 2023-04-06 17:57:14,828 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 52f45a5b42cb45ef9e824f9ecc959650, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=940, earliestPutTs=1680803834030 2023-04-06 17:57:14,828 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e50f5efba45d4e3fbc66d1e781574b35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1680803834100 2023-04-06 17:57:14,833 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#B#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:14,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:14,839 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:57:14,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:14,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:14,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:14,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:14,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:14,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:14,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803894876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803894877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680803834872, max timestamp=1680803834877Num uniques 859; 2023-04-06 17:57:14,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680803834872, max timestamp=1680803834877Num uniques 859; 2023-04-06 17:57:14,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:14,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680803834872, max timestamp=1680803834877Num uniques 859; 2023-04-06 17:57:14,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:14,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:14,896 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:14,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:14,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:14,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:14,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:14,898 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:14,898 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,899 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:14,899 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:14,899 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,899 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,901 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,901 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,901 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:14,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:14,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:14,903 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:14,903 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:14,903 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,903 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:14,904 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:14,904 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,904 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:14,904 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:14,904 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,904 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:14,905 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:14,905 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,905 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:14,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eab368b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:14,907 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:14,908 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,908 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,908 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:14,908 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:14,908 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:14,909 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,909 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:14,909 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:14,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:14,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:14,911 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:14,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:14,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:14,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:14,912 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:14,912 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:14,912 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,913 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:14,913 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,914 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803894909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:14,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:14,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:14,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:14,916 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:14,916 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:14,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:14,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 915 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803894914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803894979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:14,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803894979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:14,986 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/c284a02d8a20496e8c01bab5e49b538a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c284a02d8a20496e8c01bab5e49b538a 2023-04-06 17:57:14,989 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/A of 5c47f717ab27776f2d38f585a39f2180 into c284a02d8a20496e8c01bab5e49b538a(size=26.0 K), total size for store is 90.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:14,990 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:14,990 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/A, priority=12, startTime=1680803834050; duration=0sec 2023-04-06 17:57:14,990 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:14,990 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:14,990 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:14,990 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/C is initiating minor compaction (all files) 2023-04-06 17:57:14,991 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/C in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:14,991 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/615820e6e6eb4750a1f967408a2f67c9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/964520796ad2487c91bb25084f88caa7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/244538aa58c74bf39aa555c7f1d5a942] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=36.3 K 2023-04-06 17:57:14,991 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 615820e6e6eb4750a1f967408a2f67c9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1680803833935 2023-04-06 17:57:14,991 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 964520796ad2487c91bb25084f88caa7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=940, earliestPutTs=1680803834030 2023-04-06 17:57:14,992 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 244538aa58c74bf39aa555c7f1d5a942, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1680803834100 2023-04-06 17:57:14,997 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#C#compaction#188 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:15,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803895015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,016 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:15,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:15,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:15,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:15,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:15,019 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:15,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:15,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:15,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,020 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,020 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,020 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 917 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803895019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:15,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:15,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:15,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:15,021 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,021 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:15,021 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:15,021 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,021 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:15,022 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:15,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,022 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:15,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e934c86[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:15,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:15,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:15,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:15,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:15,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:15,025 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:15,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:15,027 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:15,027 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:15,027 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:15,027 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:15,027 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:15,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,029 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,030 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:15,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,030 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,131 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:15,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:15,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:15,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:15,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:15,133 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:15,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:15,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:15,134 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,134 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,134 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,134 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,134 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,134 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:15,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:15,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:15,135 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:15,135 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,135 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:15,135 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:15,135 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:15,135 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,136 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:15,136 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:15,136 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:15,137 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,137 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:15,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:15,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e3017dc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:15,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,138 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:15,139 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,139 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,139 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:15,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:15,141 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:15,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:15,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:15,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:15,141 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:15,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,143 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,143 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:15,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:15,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,145 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:15,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,145 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,145 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803895184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803895185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803895218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 919 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803895222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,241 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/30a357f123794642aced28b4e6812060 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30a357f123794642aced28b4e6812060 2023-04-06 17:57:15,245 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:15,245 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/B of 5c47f717ab27776f2d38f585a39f2180 into 30a357f123794642aced28b4e6812060(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:15,245 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:15,245 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/B, priority=13, startTime=1680803834825; duration=0sec 2023-04-06 17:57:15,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:15,246 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:15,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:15,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:15,248 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:15,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:15,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:15,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:15,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:15,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:15,250 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,250 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:15,250 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:15,250 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:15,252 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,252 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,252 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:15,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe6544[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:15,255 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040625f64e6babdb435badf928ce4834306f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040625f64e6babdb435badf928ce4834306f 2023-04-06 17:57:15,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:15,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,256 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,256 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:15,256 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:15,256 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:15,256 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:15,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:15,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:15,258 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:15,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:15,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:15,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:15,258 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:15,259 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,259 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,259 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,261 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=975, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/4851c30ccc5646ff9e2d24d96abfaf3e 2023-04-06 17:57:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,261 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,263 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,263 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,263 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,263 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,263 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:15,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:15,263 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:15,264 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:15,264 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:15,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=975 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/55183084df684308bcb942544242c4a7 2023-04-06 17:57:15,364 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:15,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:15,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:15,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:15,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:15,366 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:15,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:15,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:15,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,368 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:15,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:15,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:15,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:15,369 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:15,369 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,369 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:15,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:15,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:15,370 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,370 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:15,370 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:15,370 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:15,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:15,371 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,371 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:15,371 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:15,371 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,371 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,372 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,372 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:15,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6606049e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:15,373 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:15,373 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,373 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:15,374 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,374 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:15,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:15,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:15,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,375 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,375 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:15,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:15,376 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:15,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:15,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:15,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:15,376 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:15,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:15,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,380 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:15,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:15,381 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:15,381 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:15,408 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/cdab7d56a52043f3ae5e6798894ac4b6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cdab7d56a52043f3ae5e6798894ac4b6 2023-04-06 17:57:15,412 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/C of 5c47f717ab27776f2d38f585a39f2180 into cdab7d56a52043f3ae5e6798894ac4b6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:15,412 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:15,412 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/C, priority=13, startTime=1680803834825; duration=0sec 2023-04-06 17:57:15,412 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:15,481 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:15,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:15,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:15,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:15,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:15,483 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:15,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:15,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:15,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:15,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:15,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:15,486 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,486 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:15,486 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:15,486 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:15,487 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,487 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726583a4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:15,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:15,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:15,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:15,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:15,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:15,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803895488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803895488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:15,492 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:15,492 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:15,492 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:15,492 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:15,492 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:15,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,493 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,496 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,496 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,496 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,496 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:15,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,497 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:15,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,497 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803895520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:15,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 921 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803895523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,597 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:15,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:15,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:15,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:15,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:15,600 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:15,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:15,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:15,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,601 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:15,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:15,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:15,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:15,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:15,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c15e4c3[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:15,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,606 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:15,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:15,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:15,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:15,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:15,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:15,609 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:15,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:15,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:15,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:15,609 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:15,610 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,610 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:15,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,610 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:15,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,611 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:15,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:15,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,613 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:15,613 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:15,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:15,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:15,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=975 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/57812ef2dd9c44baa0c9ca1c78402704 2023-04-06 17:57:15,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/4851c30ccc5646ff9e2d24d96abfaf3e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4851c30ccc5646ff9e2d24d96abfaf3e 2023-04-06 17:57:15,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4851c30ccc5646ff9e2d24d96abfaf3e, entries=200, sequenceid=975, filesize=32.3 K 2023-04-06 17:57:15,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/55183084df684308bcb942544242c4a7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/55183084df684308bcb942544242c4a7 2023-04-06 17:57:15,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/55183084df684308bcb942544242c4a7, entries=150, sequenceid=975, filesize=11.9 K 2023-04-06 17:57:15,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/57812ef2dd9c44baa0c9ca1c78402704 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/57812ef2dd9c44baa0c9ca1c78402704 2023-04-06 17:57:15,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/57812ef2dd9c44baa0c9ca1c78402704, entries=150, sequenceid=975, filesize=11.9 K 2023-04-06 17:57:15,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 5c47f717ab27776f2d38f585a39f2180 in 863ms, sequenceid=975, compaction requested=true 2023-04-06 17:57:15,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:15,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:15,702 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:15,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:15,702 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 17:57:15,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:15,703 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:15,703 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:15,703 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. because compaction request was cancelled 2023-04-06 17:57:15,703 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 17:57:15,704 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125918 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:15,704 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/A is initiating minor compaction (all files) 2023-04-06 17:57:15,704 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/A in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,704 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c284a02d8a20496e8c01bab5e49b538a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f44d40ff9ecc4617b1c88a528467fb5a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/767b82c0dcce469c87c44785492912fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4851c30ccc5646ff9e2d24d96abfaf3e] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=123.0 K 2023-04-06 17:57:15,704 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c284a02d8a20496e8c01bab5e49b538a, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1680803833935 2023-04-06 17:57:15,704 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:15,704 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:15,704 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. because compaction request was cancelled 2023-04-06 17:57:15,705 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f44d40ff9ecc4617b1c88a528467fb5a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=940, earliestPutTs=1680803834030 2023-04-06 17:57:15,705 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 767b82c0dcce469c87c44785492912fb, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1680803834100 2023-04-06 17:57:15,705 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4851c30ccc5646ff9e2d24d96abfaf3e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680803834752 2023-04-06 17:57:15,713 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:15,714 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#A#compaction#191 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:15,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:15,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:15,716 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:15,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:15,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:15,717 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,717 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,718 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:15,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:15,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:15,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:15,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:15,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:15,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:15,719 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:15,719 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. started... 2023-04-06 17:57:15,719 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-06 17:57:15,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:15,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:15,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:15,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:15,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:15,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:15,736 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040640ae45d4ff2d4715ae8508c33e70aec4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040640ae45d4ff2d4715ae8508c33e70aec4 2023-04-06 17:57:15,742 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1012, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/f5a76de1062646c993eaadabdfd73be7 2023-04-06 17:57:15,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:15,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. as already flushing 2023-04-06 17:57:16,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803896007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803896009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680803896024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 923 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57600 deadline: 1680803896026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803896110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803896112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,150 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1012 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/4f08828b057f4848bf8bc8ed6503bfda 2023-04-06 17:57:16,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57636 deadline: 1680803896313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:16,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57630 deadline: 1680803896314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,511 DEBUG [Thread-960] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e7aae9f to 127.0.0.1:49384 2023-04-06 17:57:16,511 DEBUG [Thread-960] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:16,511 DEBUG [Thread-956] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3e8e6c to 127.0.0.1:49384 2023-04-06 17:57:16,511 DEBUG [Thread-956] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:16,513 DEBUG [Thread-958] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31c99ac6 to 127.0.0.1:49384 2023-04-06 17:57:16,513 DEBUG [Thread-958] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:16,517 DEBUG [Thread-962] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ca9d9a3 to 127.0.0.1:49384 2023-04-06 17:57:16,517 DEBUG [Thread-962] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:16,562 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1012 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/88abafdfc9e04164b87763eb02b7cba2 2023-04-06 17:57:16,571 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/f5a76de1062646c993eaadabdfd73be7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f5a76de1062646c993eaadabdfd73be7 2023-04-06 17:57:16,579 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f5a76de1062646c993eaadabdfd73be7, entries=150, sequenceid=1012, filesize=25.4 K 2023-04-06 17:57:16,580 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/4f08828b057f4848bf8bc8ed6503bfda as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/4f08828b057f4848bf8bc8ed6503bfda 2023-04-06 17:57:16,586 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/4f08828b057f4848bf8bc8ed6503bfda, entries=150, sequenceid=1012, filesize=11.9 K 2023-04-06 17:57:16,587 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/88abafdfc9e04164b87763eb02b7cba2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/88abafdfc9e04164b87763eb02b7cba2 2023-04-06 17:57:16,593 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/88abafdfc9e04164b87763eb02b7cba2, entries=150, sequenceid=1012, filesize=11.9 K 2023-04-06 17:57:16,594 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 5c47f717ab27776f2d38f585a39f2180 in 875ms, sequenceid=1012, compaction requested=true 2023-04-06 17:57:16,594 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:16,594 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:16,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:16,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:16,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:16,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:16,596 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:16,597 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,597 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:16,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:16,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb37440[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:16,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:16,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,600 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,600 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:16,600 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:16,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:16,600 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:16,601 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:16,602 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:16,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:16,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:16,602 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:16,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:16,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:16,605 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:16,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:16,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:16,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:16,606 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:16,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:16,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:16,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:16,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:16,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:16,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:16,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:16,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:57:16,618 DEBUG [Thread-953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x537a0ff8 to 127.0.0.1:49384 2023-04-06 17:57:16,618 DEBUG [Thread-953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:16,619 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,620 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:16,620 DEBUG [Thread-947] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5499dabb to 127.0.0.1:49384 2023-04-06 17:57:16,620 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:16,620 DEBUG [Thread-947] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:16,620 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:16,620 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:16,620 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:16,620 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:16,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:16,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:16,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:16,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:16,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:16,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:16,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:16,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:16,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:16,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:16,626 DEBUG [Thread-955] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:16,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:16,626 DEBUG [Thread-955] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:16,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:16,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:16,726 DEBUG [Thread-955] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:16,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:16,939 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/645fbfa1c590487ca55b7e5ca4037baf as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/645fbfa1c590487ca55b7e5ca4037baf 2023-04-06 17:57:16,945 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/A of 5c47f717ab27776f2d38f585a39f2180 into 645fbfa1c590487ca55b7e5ca4037baf(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 17:57:16,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:16,945 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/A, priority=12, startTime=1680803835702; duration=1sec 2023-04-06 17:57:16,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:17,035 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406fae3448d34d246b6badd1e334a78ee44 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406fae3448d34d246b6badd1e334a78ee44 2023-04-06 17:57:17,037 DEBUG [Thread-949] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6565606b to 127.0.0.1:49384 2023-04-06 17:57:17,038 DEBUG [Thread-949] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:17,038 DEBUG [Thread-951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65dbf2b7 to 127.0.0.1:49384 2023-04-06 17:57:17,038 DEBUG [Thread-951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:17,463 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1026, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/3851ebee82cd4e439cbd044366f31a4a 2023-04-06 17:57:17,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1026 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/f03bd4970fcb4cea83c7849e8896a7dc 2023-04-06 17:57:18,166 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-06 17:57:18,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1026 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/154e6a8f8d7049ee9e446513288d66cd 2023-04-06 17:57:18,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/3851ebee82cd4e439cbd044366f31a4a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/3851ebee82cd4e439cbd044366f31a4a 2023-04-06 17:57:18,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/3851ebee82cd4e439cbd044366f31a4a, entries=150, sequenceid=1026, filesize=25.4 K 2023-04-06 17:57:18,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/f03bd4970fcb4cea83c7849e8896a7dc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/f03bd4970fcb4cea83c7849e8896a7dc 2023-04-06 17:57:18,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/f03bd4970fcb4cea83c7849e8896a7dc, entries=150, sequenceid=1026, filesize=11.9 K 2023-04-06 17:57:18,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/154e6a8f8d7049ee9e446513288d66cd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/154e6a8f8d7049ee9e446513288d66cd 2023-04-06 17:57:18,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/154e6a8f8d7049ee9e446513288d66cd, entries=150, sequenceid=1026, filesize=11.9 K 2023-04-06 17:57:18,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=13.42 KB/13740 for 5c47f717ab27776f2d38f585a39f2180 in 1721ms, sequenceid=1026, compaction requested=true 2023-04-06 17:57:18,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:18,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:18,340 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:18,340 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:18,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:18,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:18,340 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78888 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:18,341 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/A is initiating minor compaction (all files) 2023-04-06 17:57:18,341 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/A in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:18,341 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/645fbfa1c590487ca55b7e5ca4037baf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f5a76de1062646c993eaadabdfd73be7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/3851ebee82cd4e439cbd044366f31a4a] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=77.0 K 2023-04-06 17:57:18,341 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:18,341 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/B is initiating minor compaction (all files) 2023-04-06 17:57:18,341 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/B in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:18,341 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30a357f123794642aced28b4e6812060, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/55183084df684308bcb942544242c4a7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/4f08828b057f4848bf8bc8ed6503bfda, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/f03bd4970fcb4cea83c7849e8896a7dc] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=48.3 K 2023-04-06 17:57:18,341 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 645fbfa1c590487ca55b7e5ca4037baf, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680803834752 2023-04-06 17:57:18,341 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 30a357f123794642aced28b4e6812060, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1680803834100 2023-04-06 17:57:18,341 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f5a76de1062646c993eaadabdfd73be7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1012, earliestPutTs=1680803834896 2023-04-06 17:57:18,342 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 55183084df684308bcb942544242c4a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680803834752 2023-04-06 17:57:18,342 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3851ebee82cd4e439cbd044366f31a4a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1026, earliestPutTs=1680803836008 2023-04-06 17:57:18,342 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4f08828b057f4848bf8bc8ed6503bfda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1012, earliestPutTs=1680803834896 2023-04-06 17:57:18,342 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f03bd4970fcb4cea83c7849e8896a7dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1026, earliestPutTs=1680803836008 2023-04-06 17:57:18,350 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#B#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:18,353 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#A#compaction#199 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:18,772 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/d21dca61d4a44787944a02c783d4305b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d21dca61d4a44787944a02c783d4305b 2023-04-06 17:57:18,778 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/B of 5c47f717ab27776f2d38f585a39f2180 into d21dca61d4a44787944a02c783d4305b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:18,778 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:18,778 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/B, priority=12, startTime=1680803838340; duration=0sec 2023-04-06 17:57:18,778 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:18,778 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:18,780 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:18,780 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 5c47f717ab27776f2d38f585a39f2180/C is initiating minor compaction (all files) 2023-04-06 17:57:18,780 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c47f717ab27776f2d38f585a39f2180/C in TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:18,780 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cdab7d56a52043f3ae5e6798894ac4b6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/57812ef2dd9c44baa0c9ca1c78402704, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/88abafdfc9e04164b87763eb02b7cba2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/154e6a8f8d7049ee9e446513288d66cd] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp, totalSize=48.3 K 2023-04-06 17:57:18,781 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cdab7d56a52043f3ae5e6798894ac4b6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1680803834100 2023-04-06 17:57:18,781 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 57812ef2dd9c44baa0c9ca1c78402704, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680803834752 2023-04-06 17:57:18,781 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 88abafdfc9e04164b87763eb02b7cba2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1012, earliestPutTs=1680803834896 2023-04-06 17:57:18,782 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 154e6a8f8d7049ee9e446513288d66cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1026, earliestPutTs=1680803836008 2023-04-06 17:57:18,793 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c47f717ab27776f2d38f585a39f2180#C#compaction#200 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:19,185 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/e3b4c81deb664e129633b6b059dafdcf as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e3b4c81deb664e129633b6b059dafdcf 2023-04-06 17:57:19,193 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/A of 5c47f717ab27776f2d38f585a39f2180 into e3b4c81deb664e129633b6b059dafdcf(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:19,193 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:19,193 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/A, priority=13, startTime=1680803838339; duration=0sec 2023-04-06 17:57:19,193 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:19,206 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/29faba8e2679445ba01ad8fe8eadc263 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/29faba8e2679445ba01ad8fe8eadc263 2023-04-06 17:57:19,213 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c47f717ab27776f2d38f585a39f2180/C of 5c47f717ab27776f2d38f585a39f2180 into 29faba8e2679445ba01ad8fe8eadc263(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:19,213 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:19,213 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180., storeName=5c47f717ab27776f2d38f585a39f2180/C, priority=12, startTime=1680803838340; duration=0sec 2023-04-06 17:57:19,213 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:20,956 DEBUG [Thread-945] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13522ca3 to 127.0.0.1:49384 2023-04-06 17:57:20,956 DEBUG [Thread-945] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 828 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 5524 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 5376 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2349 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 7047 rows 2023-04-06 17:57:20,957 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2353 2023-04-06 17:57:20,958 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 7059 rows 2023-04-06 17:57:20,958 INFO [Listener at localhost.localdomain/40169] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 17:57:20,958 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3efe2ee2 to 127.0.0.1:49384 2023-04-06 17:57:20,958 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:20,959 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 17:57:20,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 17:57:20,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:20,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 17:57:20,964 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803840964"}]},"ts":"1680803840964"} 2023-04-06 17:57:20,965 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 17:57:21,061 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 17:57:21,062 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, UNASSIGN}] 2023-04-06 17:57:21,063 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, UNASSIGN 2023-04-06 17:57:21,064 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 17:57:21,065 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803841064"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803841064"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803841064"}]},"ts":"1680803841064"} 2023-04-06 17:57:21,066 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:57:21,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:21,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 5c47f717ab27776f2d38f585a39f2180, disabling compactions & flushes 2023-04-06 17:57:21,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:21,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:21,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. after waiting 0 ms 2023-04-06 17:57:21,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:21,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 5c47f717ab27776f2d38f585a39f2180 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 17:57:21,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=A 2023-04-06 17:57:21,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:21,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=B 2023-04-06 17:57:21,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:21,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c47f717ab27776f2d38f585a39f2180, store=C 2023-04-06 17:57:21,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:21,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 17:57:21,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 17:57:21,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406cb14d63aace447d28652760a7f0b3c7e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cb14d63aace447d28652760a7f0b3c7e 2023-04-06 17:57:22,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1036, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/880e9a64ac924b0d8c36b0c912485bb4 2023-04-06 17:57:22,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 17:57:22,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/2afb440958ed4664832d163d24d13fe5 2023-04-06 17:57:22,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/233e1cfef2fc40e8857b634d31a93281 2023-04-06 17:57:22,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/A/880e9a64ac924b0d8c36b0c912485bb4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/880e9a64ac924b0d8c36b0c912485bb4 2023-04-06 17:57:22,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/880e9a64ac924b0d8c36b0c912485bb4, entries=100, sequenceid=1036, filesize=18.6 K 2023-04-06 17:57:22,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/B/2afb440958ed4664832d163d24d13fe5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2afb440958ed4664832d163d24d13fe5 2023-04-06 17:57:22,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2afb440958ed4664832d163d24d13fe5, entries=100, sequenceid=1036, filesize=9.5 K 2023-04-06 17:57:22,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/.tmp/C/233e1cfef2fc40e8857b634d31a93281 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/233e1cfef2fc40e8857b634d31a93281 2023-04-06 17:57:22,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/233e1cfef2fc40e8857b634d31a93281, entries=100, sequenceid=1036, filesize=9.5 K 2023-04-06 17:57:22,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 5c47f717ab27776f2d38f585a39f2180 in 1691ms, sequenceid=1036, compaction requested=false 2023-04-06 17:57:22,932 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/687f0f570e004332ad5bc68432cc72cc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/9c09a9830beb44bcbd72047a841790fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/a4960e91d22c4c63a1bee86bd46f59b6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/17a9ca0dea814f19b6d7b661bffc101d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/ac94daa7fda34233a5168fd48ab8e2db, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b2d76d3843444befbe7265a1d50a276a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/782e8a4d492f44648aa5b1a86983da00, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e4e6fe65452a46d5970f8a4e105c2aa6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4bcbae3b8d1f408cb8d3bf37de7267e9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/bb75e3d003fc4b31af3df7584cf6de6a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/0e9e40ba210741e3bbcd9395972a40c4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b96efcf3ad71430da0232aabca43e6eb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c9240783d37f45a1b46984cb22425705, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b20a2ac5d4794a7f871a447e6edb6053, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02cbeed2a1ab474484e2e636359cafae, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/093864af92634efead19ed61963de794, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/df81ab24bab24d7880cb613bc367af7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02fe5a72d7404ecaaed1bec8911c7d70, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c284a02d8a20496e8c01bab5e49b538a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f44d40ff9ecc4617b1c88a528467fb5a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/767b82c0dcce469c87c44785492912fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4851c30ccc5646ff9e2d24d96abfaf3e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/645fbfa1c590487ca55b7e5ca4037baf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f5a76de1062646c993eaadabdfd73be7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/3851ebee82cd4e439cbd044366f31a4a] to archive 2023-04-06 17:57:22,933 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:57:22,935 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/687f0f570e004332ad5bc68432cc72cc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/687f0f570e004332ad5bc68432cc72cc 2023-04-06 17:57:22,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/9c09a9830beb44bcbd72047a841790fb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/9c09a9830beb44bcbd72047a841790fb 2023-04-06 17:57:22,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/a4960e91d22c4c63a1bee86bd46f59b6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/a4960e91d22c4c63a1bee86bd46f59b6 2023-04-06 17:57:22,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/17a9ca0dea814f19b6d7b661bffc101d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/17a9ca0dea814f19b6d7b661bffc101d 2023-04-06 17:57:22,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/ac94daa7fda34233a5168fd48ab8e2db to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/ac94daa7fda34233a5168fd48ab8e2db 2023-04-06 17:57:22,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b2d76d3843444befbe7265a1d50a276a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b2d76d3843444befbe7265a1d50a276a 2023-04-06 17:57:22,942 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/782e8a4d492f44648aa5b1a86983da00 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/782e8a4d492f44648aa5b1a86983da00 2023-04-06 17:57:22,943 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e4e6fe65452a46d5970f8a4e105c2aa6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e4e6fe65452a46d5970f8a4e105c2aa6 2023-04-06 17:57:22,943 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4bcbae3b8d1f408cb8d3bf37de7267e9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4bcbae3b8d1f408cb8d3bf37de7267e9 2023-04-06 17:57:22,944 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/bb75e3d003fc4b31af3df7584cf6de6a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/bb75e3d003fc4b31af3df7584cf6de6a 2023-04-06 17:57:22,945 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/0e9e40ba210741e3bbcd9395972a40c4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/0e9e40ba210741e3bbcd9395972a40c4 2023-04-06 17:57:22,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b96efcf3ad71430da0232aabca43e6eb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b96efcf3ad71430da0232aabca43e6eb 2023-04-06 17:57:22,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c9240783d37f45a1b46984cb22425705 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c9240783d37f45a1b46984cb22425705 2023-04-06 17:57:22,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b20a2ac5d4794a7f871a447e6edb6053 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/b20a2ac5d4794a7f871a447e6edb6053 2023-04-06 17:57:22,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02cbeed2a1ab474484e2e636359cafae to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02cbeed2a1ab474484e2e636359cafae 2023-04-06 17:57:22,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/093864af92634efead19ed61963de794 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/093864af92634efead19ed61963de794 2023-04-06 17:57:22,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/df81ab24bab24d7880cb613bc367af7a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/df81ab24bab24d7880cb613bc367af7a 2023-04-06 17:57:22,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02fe5a72d7404ecaaed1bec8911c7d70 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/02fe5a72d7404ecaaed1bec8911c7d70 2023-04-06 17:57:22,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c284a02d8a20496e8c01bab5e49b538a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/c284a02d8a20496e8c01bab5e49b538a 2023-04-06 17:57:22,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f44d40ff9ecc4617b1c88a528467fb5a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f44d40ff9ecc4617b1c88a528467fb5a 2023-04-06 17:57:22,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/767b82c0dcce469c87c44785492912fb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/767b82c0dcce469c87c44785492912fb 2023-04-06 17:57:22,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4851c30ccc5646ff9e2d24d96abfaf3e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/4851c30ccc5646ff9e2d24d96abfaf3e 2023-04-06 17:57:22,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/645fbfa1c590487ca55b7e5ca4037baf to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/645fbfa1c590487ca55b7e5ca4037baf 2023-04-06 17:57:22,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f5a76de1062646c993eaadabdfd73be7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/f5a76de1062646c993eaadabdfd73be7 2023-04-06 17:57:22,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/3851ebee82cd4e439cbd044366f31a4a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/3851ebee82cd4e439cbd044366f31a4a 2023-04-06 17:57:22,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/26e7ad86920547fb853acaf7a83df506, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9dcf67b54bd64abd9c0308079afcb654, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/a25bd9e03a8a4f14b85080aaa7bc0b26, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2dcd4bd4b6754af3ad46044680fce1ea, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d24be39dd5bb4563b6759ed2bd26a1fa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/29e70b1fc9d5453dba4b0341163888b1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dc248fb07e374d478fdb9eb1b105ccfa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/3a0096183aa34b57837e2e5ac874479a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30fe055e74d64b2d95d45009c64a31ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9d72825cb7304f7d9f83a91917ad894b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/eafade9dd1f841239e57f0a74ebc68b6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/b94050c737664ab79f4aee4d40bbfdcc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cfe03e9fb55245608c2b136d5f776c08, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/5933947a45c546579582e95decaeb5fd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cc927a650a3146f1bd29b28850c4befd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dec698bc3fbc4448a5802d18e1591b6c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/8e8c87eb46a94e33a564323bf0e00126, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/fc94cacc99e54b3c9b4af7bf8309b11d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/0c18d8cf32e34546b3240e26666a8934, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/52f45a5b42cb45ef9e824f9ecc959650, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30a357f123794642aced28b4e6812060, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/e50f5efba45d4e3fbc66d1e781574b35, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/55183084df684308bcb942544242c4a7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/4f08828b057f4848bf8bc8ed6503bfda, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/f03bd4970fcb4cea83c7849e8896a7dc] to archive 2023-04-06 17:57:22,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:57:22,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/26e7ad86920547fb853acaf7a83df506 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/26e7ad86920547fb853acaf7a83df506 2023-04-06 17:57:22,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9dcf67b54bd64abd9c0308079afcb654 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9dcf67b54bd64abd9c0308079afcb654 2023-04-06 17:57:22,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/a25bd9e03a8a4f14b85080aaa7bc0b26 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/a25bd9e03a8a4f14b85080aaa7bc0b26 2023-04-06 17:57:22,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2dcd4bd4b6754af3ad46044680fce1ea to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2dcd4bd4b6754af3ad46044680fce1ea 2023-04-06 17:57:22,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d24be39dd5bb4563b6759ed2bd26a1fa to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d24be39dd5bb4563b6759ed2bd26a1fa 2023-04-06 17:57:22,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/29e70b1fc9d5453dba4b0341163888b1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/29e70b1fc9d5453dba4b0341163888b1 2023-04-06 17:57:22,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dc248fb07e374d478fdb9eb1b105ccfa to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dc248fb07e374d478fdb9eb1b105ccfa 2023-04-06 17:57:22,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/3a0096183aa34b57837e2e5ac874479a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/3a0096183aa34b57837e2e5ac874479a 2023-04-06 17:57:22,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30fe055e74d64b2d95d45009c64a31ab to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30fe055e74d64b2d95d45009c64a31ab 2023-04-06 17:57:22,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9d72825cb7304f7d9f83a91917ad894b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/9d72825cb7304f7d9f83a91917ad894b 2023-04-06 17:57:22,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/eafade9dd1f841239e57f0a74ebc68b6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/eafade9dd1f841239e57f0a74ebc68b6 2023-04-06 17:57:22,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/b94050c737664ab79f4aee4d40bbfdcc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/b94050c737664ab79f4aee4d40bbfdcc 2023-04-06 17:57:22,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cfe03e9fb55245608c2b136d5f776c08 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cfe03e9fb55245608c2b136d5f776c08 2023-04-06 17:57:22,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/5933947a45c546579582e95decaeb5fd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/5933947a45c546579582e95decaeb5fd 2023-04-06 17:57:22,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cc927a650a3146f1bd29b28850c4befd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/cc927a650a3146f1bd29b28850c4befd 2023-04-06 17:57:22,976 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dec698bc3fbc4448a5802d18e1591b6c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/dec698bc3fbc4448a5802d18e1591b6c 2023-04-06 17:57:22,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/8e8c87eb46a94e33a564323bf0e00126 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/8e8c87eb46a94e33a564323bf0e00126 2023-04-06 17:57:22,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/fc94cacc99e54b3c9b4af7bf8309b11d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/fc94cacc99e54b3c9b4af7bf8309b11d 2023-04-06 17:57:22,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/0c18d8cf32e34546b3240e26666a8934 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/0c18d8cf32e34546b3240e26666a8934 2023-04-06 17:57:22,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/52f45a5b42cb45ef9e824f9ecc959650 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/52f45a5b42cb45ef9e824f9ecc959650 2023-04-06 17:57:22,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30a357f123794642aced28b4e6812060 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/30a357f123794642aced28b4e6812060 2023-04-06 17:57:22,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/e50f5efba45d4e3fbc66d1e781574b35 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/e50f5efba45d4e3fbc66d1e781574b35 2023-04-06 17:57:22,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/55183084df684308bcb942544242c4a7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/55183084df684308bcb942544242c4a7 2023-04-06 17:57:22,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/4f08828b057f4848bf8bc8ed6503bfda to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/4f08828b057f4848bf8bc8ed6503bfda 2023-04-06 17:57:22,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/f03bd4970fcb4cea83c7849e8896a7dc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/f03bd4970fcb4cea83c7849e8896a7dc 2023-04-06 17:57:22,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/066af0f2852048b1847f897587ff08d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4473a68fcfb34a27b1adca610c543f98, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/5c21f0ca9ac94fe9a6d13db842307852, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/8b8f473920a64f849d07507bc95d904f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/2db9d70140534c038d42c3f8dc744717, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/103bd906bab2496cbd0470e4d3e0e696, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/1cfd6c957caf4fa58a37f8142f645151, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/276aade37b6d46ba9e3d17baced1a96d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/ff35fc0e180c4b408e67c0adb1e287aa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cd96008bd381462e9aca5cac1dae82f4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4365a22004064165a7a6ab2fbfa456fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/0f8158a0f392478985950b871fbc3d61, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/28352f417eb24bcdbba60d82c3a6fec8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/c4ac5bdcc8f44c27983c5f30ccdc7fe7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e0214068f0ea43eabcecd575d5e53ded, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e42bcdcbe56746b0ba68b8038304eaac, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/17def39147bc4681a82374a0d804f42c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/615820e6e6eb4750a1f967408a2f67c9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/f8829652842a4a8e9eda9af0895a7958, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/964520796ad2487c91bb25084f88caa7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cdab7d56a52043f3ae5e6798894ac4b6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/244538aa58c74bf39aa555c7f1d5a942, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/57812ef2dd9c44baa0c9ca1c78402704, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/88abafdfc9e04164b87763eb02b7cba2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/154e6a8f8d7049ee9e446513288d66cd] to archive 2023-04-06 17:57:22,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:57:22,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/066af0f2852048b1847f897587ff08d2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/066af0f2852048b1847f897587ff08d2 2023-04-06 17:57:22,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4473a68fcfb34a27b1adca610c543f98 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4473a68fcfb34a27b1adca610c543f98 2023-04-06 17:57:22,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/5c21f0ca9ac94fe9a6d13db842307852 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/5c21f0ca9ac94fe9a6d13db842307852 2023-04-06 17:57:22,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/8b8f473920a64f849d07507bc95d904f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/8b8f473920a64f849d07507bc95d904f 2023-04-06 17:57:22,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/2db9d70140534c038d42c3f8dc744717 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/2db9d70140534c038d42c3f8dc744717 2023-04-06 17:57:22,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/103bd906bab2496cbd0470e4d3e0e696 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/103bd906bab2496cbd0470e4d3e0e696 2023-04-06 17:57:22,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/1cfd6c957caf4fa58a37f8142f645151 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/1cfd6c957caf4fa58a37f8142f645151 2023-04-06 17:57:22,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/276aade37b6d46ba9e3d17baced1a96d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/276aade37b6d46ba9e3d17baced1a96d 2023-04-06 17:57:22,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/ff35fc0e180c4b408e67c0adb1e287aa to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/ff35fc0e180c4b408e67c0adb1e287aa 2023-04-06 17:57:22,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cd96008bd381462e9aca5cac1dae82f4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cd96008bd381462e9aca5cac1dae82f4 2023-04-06 17:57:22,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4365a22004064165a7a6ab2fbfa456fb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/4365a22004064165a7a6ab2fbfa456fb 2023-04-06 17:57:22,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/0f8158a0f392478985950b871fbc3d61 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/0f8158a0f392478985950b871fbc3d61 2023-04-06 17:57:22,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/28352f417eb24bcdbba60d82c3a6fec8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/28352f417eb24bcdbba60d82c3a6fec8 2023-04-06 17:57:23,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/c4ac5bdcc8f44c27983c5f30ccdc7fe7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/c4ac5bdcc8f44c27983c5f30ccdc7fe7 2023-04-06 17:57:23,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e0214068f0ea43eabcecd575d5e53ded to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e0214068f0ea43eabcecd575d5e53ded 2023-04-06 17:57:23,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e42bcdcbe56746b0ba68b8038304eaac to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/e42bcdcbe56746b0ba68b8038304eaac 2023-04-06 17:57:23,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/17def39147bc4681a82374a0d804f42c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/17def39147bc4681a82374a0d804f42c 2023-04-06 17:57:23,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/615820e6e6eb4750a1f967408a2f67c9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/615820e6e6eb4750a1f967408a2f67c9 2023-04-06 17:57:23,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/f8829652842a4a8e9eda9af0895a7958 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/f8829652842a4a8e9eda9af0895a7958 2023-04-06 17:57:23,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/964520796ad2487c91bb25084f88caa7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/964520796ad2487c91bb25084f88caa7 2023-04-06 17:57:23,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cdab7d56a52043f3ae5e6798894ac4b6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/cdab7d56a52043f3ae5e6798894ac4b6 2023-04-06 17:57:23,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/244538aa58c74bf39aa555c7f1d5a942 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/244538aa58c74bf39aa555c7f1d5a942 2023-04-06 17:57:23,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/57812ef2dd9c44baa0c9ca1c78402704 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/57812ef2dd9c44baa0c9ca1c78402704 2023-04-06 17:57:23,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/88abafdfc9e04164b87763eb02b7cba2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/88abafdfc9e04164b87763eb02b7cba2 2023-04-06 17:57:23,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/154e6a8f8d7049ee9e446513288d66cd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/154e6a8f8d7049ee9e446513288d66cd 2023-04-06 17:57:23,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/recovered.edits/1039.seqid, newMaxSeqId=1039, maxSeqId=4 2023-04-06 17:57:23,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180. 2023-04-06 17:57:23,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 5c47f717ab27776f2d38f585a39f2180: 2023-04-06 17:57:23,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:23,017 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=5c47f717ab27776f2d38f585a39f2180, regionState=CLOSED 2023-04-06 17:57:23,018 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803843017"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803843017"}]},"ts":"1680803843017"} 2023-04-06 17:57:23,020 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-06 17:57:23,020 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 5c47f717ab27776f2d38f585a39f2180, server=jenkins-hbase20.apache.org,38773,1680803776439 in 1.9530 sec 2023-04-06 17:57:23,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-06 17:57:23,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c47f717ab27776f2d38f585a39f2180, UNASSIGN in 1.9580 sec 2023-04-06 17:57:23,022 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803843022"}]},"ts":"1680803843022"} 2023-04-06 17:57:23,023 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 17:57:23,024 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 17:57:23,026 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0660 sec 2023-04-06 17:57:23,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 17:57:23,070 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-06 17:57:23,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 17:57:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:23,073 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:23,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-06 17:57:23,074 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:23,078 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:23,080 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/recovered.edits] 2023-04-06 17:57:23,083 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/880e9a64ac924b0d8c36b0c912485bb4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/880e9a64ac924b0d8c36b0c912485bb4 2023-04-06 17:57:23,084 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e3b4c81deb664e129633b6b059dafdcf to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/A/e3b4c81deb664e129633b6b059dafdcf 2023-04-06 17:57:23,086 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2afb440958ed4664832d163d24d13fe5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/2afb440958ed4664832d163d24d13fe5 2023-04-06 17:57:23,087 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d21dca61d4a44787944a02c783d4305b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/B/d21dca61d4a44787944a02c783d4305b 2023-04-06 17:57:23,089 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/233e1cfef2fc40e8857b634d31a93281 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/233e1cfef2fc40e8857b634d31a93281 2023-04-06 17:57:23,090 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/29faba8e2679445ba01ad8fe8eadc263 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/C/29faba8e2679445ba01ad8fe8eadc263 2023-04-06 17:57:23,093 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/recovered.edits/1039.seqid to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180/recovered.edits/1039.seqid 2023-04-06 17:57:23,093 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/5c47f717ab27776f2d38f585a39f2180 2023-04-06 17:57:23,094 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:57:23,094 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 17:57:23,095 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 17:57:23,098 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040602a3d046169440ed92da3790853a6b17 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040602a3d046169440ed92da3790853a6b17 2023-04-06 17:57:23,100 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406054886b3e6b54bacba00676f8a3f7fb9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406054886b3e6b54bacba00676f8a3f7fb9 2023-04-06 17:57:23,101 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040625f64e6babdb435badf928ce4834306f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040625f64e6babdb435badf928ce4834306f 2023-04-06 17:57:23,102 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040640ae45d4ff2d4715ae8508c33e70aec4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040640ae45d4ff2d4715ae8508c33e70aec4 2023-04-06 17:57:23,103 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040649e6ff62c7d3410291ce27842e1ba9d3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040649e6ff62c7d3410291ce27842e1ba9d3 2023-04-06 17:57:23,104 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406719f2470084345c89857f6eb36dcb90a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406719f2470084345c89857f6eb36dcb90a 2023-04-06 17:57:23,105 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067b568af4d3b744dab242932276fe1732 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067b568af4d3b744dab242932276fe1732 2023-04-06 17:57:23,106 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406804e33983ccd4fb1a23468fb92b864c8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406804e33983ccd4fb1a23468fb92b864c8 2023-04-06 17:57:23,107 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406897af3a4a1b2424184fa3d51febc18e1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406897af3a4a1b2424184fa3d51febc18e1 2023-04-06 17:57:23,108 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068afcc0103078448c9894bef31fd8477a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068afcc0103078448c9894bef31fd8477a 2023-04-06 17:57:23,109 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068b46d8dd4dd146ed9b2bb2a1a814b054 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068b46d8dd4dd146ed9b2bb2a1a814b054 2023-04-06 17:57:23,110 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068d8255d3fd774c91b398f929e2e96874 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068d8255d3fd774c91b398f929e2e96874 2023-04-06 17:57:23,111 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068d8df3558f074214bc367d28ef657fc9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068d8df3558f074214bc367d28ef657fc9 2023-04-06 17:57:23,112 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069545ed7bd46c462c80c6dabb140cffc2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069545ed7bd46c462c80c6dabb140cffc2 2023-04-06 17:57:23,113 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a1c53537100f4556bdd79ecb441b2e09 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a1c53537100f4556bdd79ecb441b2e09 2023-04-06 17:57:23,114 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b8010c06e2f54019bddbef1b499ddc31 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b8010c06e2f54019bddbef1b499ddc31 2023-04-06 17:57:23,115 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cb14d63aace447d28652760a7f0b3c7e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cb14d63aace447d28652760a7f0b3c7e 2023-04-06 17:57:23,116 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406dcec363139f64aedb98b6b03f9f03b2e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406dcec363139f64aedb98b6b03f9f03b2e 2023-04-06 17:57:23,117 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406dedd1e1c54484c158141be0b5252e150 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406dedd1e1c54484c158141be0b5252e150 2023-04-06 17:57:23,118 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e9e5e2c2885a43e4bc1a5eac791126b3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e9e5e2c2885a43e4bc1a5eac791126b3 2023-04-06 17:57:23,119 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406fae3448d34d246b6badd1e334a78ee44 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406fae3448d34d246b6badd1e334a78ee44 2023-04-06 17:57:23,119 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 17:57:23,121 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:23,123 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 17:57:23,125 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 17:57:23,126 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:23,126 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 17:57:23,126 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680803843126"}]},"ts":"9223372036854775807"} 2023-04-06 17:57:23,128 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 17:57:23,128 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5c47f717ab27776f2d38f585a39f2180, NAME => 'TestAcidGuarantees,,1680803813352.5c47f717ab27776f2d38f585a39f2180.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 17:57:23,128 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 17:57:23,128 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680803843128"}]},"ts":"9223372036854775807"} 2023-04-06 17:57:23,129 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 17:57:23,131 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:23,132 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 60 msec 2023-04-06 17:57:23,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-06 17:57:23,175 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-06 17:57:23,187 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 229), OpenFileDescriptor=559 (was 542) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=326 (was 334), ProcessCount=166 (was 166), AvailableMemoryMB=2968 (was 3139) 2023-04-06 17:57:23,196 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=326, ProcessCount=166, AvailableMemoryMB=2967 2023-04-06 17:57:23,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:57:23,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:57:23,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:23,202 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 17:57:23,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-06 17:57:23,203 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 17:57:23,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 17:57:23,204 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:23,205 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05 empty. 2023-04-06 17:57:23,205 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:23,205 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:57:23,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 17:57:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 17:57:23,613 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:57:23,614 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9e540ddc475e6cca2398e6140b9c8e05, NAME => 'TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp 2023-04-06 17:57:23,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 17:57:24,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:57:24,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9e540ddc475e6cca2398e6140b9c8e05, disabling compactions & flushes 2023-04-06 17:57:24,022 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. after waiting 0 ms 2023-04-06 17:57:24,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,022 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:24,026 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 17:57:24,027 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803844027"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803844027"}]},"ts":"1680803844027"} 2023-04-06 17:57:24,029 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 17:57:24,030 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 17:57:24,030 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803844030"}]},"ts":"1680803844030"} 2023-04-06 17:57:24,031 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 17:57:24,034 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9e540ddc475e6cca2398e6140b9c8e05, ASSIGN}] 2023-04-06 17:57:24,035 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9e540ddc475e6cca2398e6140b9c8e05, ASSIGN 2023-04-06 17:57:24,036 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9e540ddc475e6cca2398e6140b9c8e05, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=false 2023-04-06 17:57:24,188 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=9e540ddc475e6cca2398e6140b9c8e05, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:24,188 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803844188"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803844188"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803844188"}]},"ts":"1680803844188"} 2023-04-06 17:57:24,191 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:57:24,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 17:57:24,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9e540ddc475e6cca2398e6140b9c8e05, NAME => 'TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:57:24,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:57:24,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,356 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,358 DEBUG [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A 2023-04-06 17:57:24,358 DEBUG [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A 2023-04-06 17:57:24,359 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:24,359 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9e540ddc475e6cca2398e6140b9c8e05 columnFamilyName A 2023-04-06 17:57:24,359 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.HStore(310): Store=9e540ddc475e6cca2398e6140b9c8e05/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:24,360 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,361 DEBUG [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B 2023-04-06 17:57:24,361 DEBUG [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B 2023-04-06 17:57:24,361 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:24,361 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9e540ddc475e6cca2398e6140b9c8e05 columnFamilyName B 2023-04-06 17:57:24,361 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.HStore(310): Store=9e540ddc475e6cca2398e6140b9c8e05/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:24,362 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,363 DEBUG [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C 2023-04-06 17:57:24,363 DEBUG [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C 2023-04-06 17:57:24,363 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:24,363 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9e540ddc475e6cca2398e6140b9c8e05 columnFamilyName C 2023-04-06 17:57:24,364 INFO [StoreOpener-9e540ddc475e6cca2398e6140b9c8e05-1] regionserver.HStore(310): Store=9e540ddc475e6cca2398e6140b9c8e05/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:24,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:57:24,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:24,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:57:24,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 9e540ddc475e6cca2398e6140b9c8e05; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72166620, jitterRate=0.07536643743515015}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:57:24,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:24,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., pid=30, masterSystemTime=1680803844344 2023-04-06 17:57:24,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:24,372 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=9e540ddc475e6cca2398e6140b9c8e05, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:24,372 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803844372"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803844372"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803844372"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803844372"}]},"ts":"1680803844372"} 2023-04-06 17:57:24,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-06 17:57:24,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 in 182 msec 2023-04-06 17:57:24,375 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-06 17:57:24,375 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9e540ddc475e6cca2398e6140b9c8e05, ASSIGN in 341 msec 2023-04-06 17:57:24,376 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 17:57:24,376 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803844376"}]},"ts":"1680803844376"} 2023-04-06 17:57:24,377 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 17:57:24,378 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 17:57:24,379 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1790 sec 2023-04-06 17:57:25,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 17:57:25,309 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-06 17:57:25,310 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3a2832f4 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,315 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bbc04f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,316 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,317 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,318 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 17:57:25,321 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 17:57:25,322 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x05fbdc11 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,329 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ebff1c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,329 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x6298fe55 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,333 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3958bd43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,333 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x5620345b to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,340 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73ef029c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,341 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x76e08050 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,344 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2135e03b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,345 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x2dbba21f to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,348 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f2b45f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,348 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x24e1e878 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,352 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21e07d99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,353 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x7e9921a7 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,365 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5590a424, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,366 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x5ffcf146 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,374 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68df7707, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,374 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3fba9cd6 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,383 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7742ca6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,384 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x5e897a83 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:25,407 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@353e11a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:25,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:25,474 DEBUG [hconnection-0x6b113353-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,474 DEBUG [hconnection-0x1006c26f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,474 DEBUG [hconnection-0x426f71d5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,474 DEBUG [hconnection-0x662d658f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,476 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,483 DEBUG [hconnection-0x48146fa9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,483 DEBUG [hconnection-0x3d2e308f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,490 DEBUG [hconnection-0x34bcc760-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,490 DEBUG [hconnection-0x3b77d8d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,491 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,485 DEBUG [hconnection-0x2026169c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,496 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,496 DEBUG [hconnection-0x32c4b190-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:25,506 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,506 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,507 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,508 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47010, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,508 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,508 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,509 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:25,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:25,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:25,513 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:25,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:25,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:25,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:25,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,515 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:25,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:25,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:25,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:25,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:25,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:25,522 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:25,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:25,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:25,531 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:25,532 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:25,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:25,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:57:25,538 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:25,538 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:25,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:25,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:25,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:25,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:25,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:25,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:25,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:25,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:25,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:25,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:25,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,541 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,541 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:25,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c7adfc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:25,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:25,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:25,544 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:25,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:25,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:25,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:25,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:25,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:25,545 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:25,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:25,548 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:25,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:25,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:25,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:25,548 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:25,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:25,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:25,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:25,555 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:25,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:25,555 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:25,555 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:25,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:25,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803905555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803905555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680803845551, max timestamp=1680803845555Num uniques 859; 2023-04-06 17:57:25,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:25,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803845551, max timestamp=1680803845555Num uniques 859; 2023-04-06 17:57:25,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:57:25,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803845551, max timestamp=1680803845555Num uniques 859; 2023-04-06 17:57:25,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:57:25,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803905581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803905581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803905594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,655 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:25,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:25,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:25,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803905658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:25,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:25,663 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:25,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:25,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:25,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:25,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803905658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,664 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,664 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:25,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:25,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:25,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:25,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:25,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:25,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:25,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:25,684 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:25,684 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:25,684 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:25,684 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:25,684 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:25,685 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:25,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:25,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:25,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:25,685 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:25,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,686 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803905682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,686 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,687 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,687 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:25,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766173e3[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:25,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:25,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,689 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:25,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:25,690 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,690 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:25,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:25,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:25,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:25,691 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:25,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:25,694 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:25,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:25,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:25,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:25,694 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:25,695 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,695 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:25,695 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:25,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803905692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803905700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,701 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,708 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:25,708 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:25,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:25,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:25,708 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:25,709 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:25,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:25,709 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:25,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:25,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:25,809 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:25,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:25,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:25,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:25,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:25,813 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:25,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:25,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:25,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:25,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,816 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:25,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:25,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:25,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:25,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:25,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:25,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:25,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:25,857 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:25,857 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:25,862 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:25,862 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:25,862 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:25,862 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:25,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:25,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:25,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:25,862 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:25,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,863 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803905861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:25,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55f1a210[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:25,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:25,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:25,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:25,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:25,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:25,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803905865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:25,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:25,869 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:25,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:25,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:25,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:25,870 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:25,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:25,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:25,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:25,872 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:25,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:25,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:25,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:25,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:25,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:25,876 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:25,876 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:25,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:25,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803905888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803905899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:25,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803905902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,976 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:25,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:25,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:25,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:25,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:25,994 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:25,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:25,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:25,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:25,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:25,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:25,995 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:25,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:25,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:26,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:26,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:26,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:26,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:26,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:26,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:26,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:26,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/1ae9c6ceb8494db28933a1e319d8c25f 2023-04-06 17:57:26,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:26,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:26,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:26,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:26,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3005cd9a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:26,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:26,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,031 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,031 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:26,031 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:26,031 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:26,031 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:26,033 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:26,033 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,033 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,033 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:26,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:26,035 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:26,036 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:26,036 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:26,036 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:26,036 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:26,037 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,037 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,037 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,038 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,040 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,049 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:26,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,050 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:26,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,050 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:26,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,053 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,150 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:26,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:26,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:26,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803906168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803906171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:26,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:26,182 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:26,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:26,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:26,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,182 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,183 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,183 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,184 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,184 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:26,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:26,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:26,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:26,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,187 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:26,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803906192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:26,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:26,203 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,203 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:26,203 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:26,203 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:26,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:26,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:26,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:26,205 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,205 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,205 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:26,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803906204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803906207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,205 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:26,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b83e2b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:26,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:26,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,223 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:26,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,224 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:26,224 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:26,224 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:26,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:26,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:26,239 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:26,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:26,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:26,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:26,239 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:26,241 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,247 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:26,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:26,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:26,250 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:26,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,250 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,250 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,350 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:26,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:26,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:26,385 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:26,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:26,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:26,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:26,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:26,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:26,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:26,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:26,409 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:26,409 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:26,410 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,410 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:26,410 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:26,410 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:26,411 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,411 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,411 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:26,423 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:26,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58bbbf14[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:26,451 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:26,451 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:26,452 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,453 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:26,453 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:26,453 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:26,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:26,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:26,459 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:26,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:26,461 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:26,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:26,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:26,462 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:26,462 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:26,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,468 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,473 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,473 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,473 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,473 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:26,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:26,473 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,473 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,473 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,474 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:26,474 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,474 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,474 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,474 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/d17e3748c1a841069429b0097a9cb120 2023-04-06 17:57:26,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/aaaa6fee0e144465a7eef4e0fd4cdd30 2023-04-06 17:57:26,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/1ae9c6ceb8494db28933a1e319d8c25f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ae9c6ceb8494db28933a1e319d8c25f 2023-04-06 17:57:26,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ae9c6ceb8494db28933a1e319d8c25f, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 17:57:26,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/d17e3748c1a841069429b0097a9cb120 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d17e3748c1a841069429b0097a9cb120 2023-04-06 17:57:26,574 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:26,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:26,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:26,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:26,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:26,588 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:26,588 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,588 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:26,588 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:26,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,590 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:26,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:26,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:26,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:26,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:26,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:26,592 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:26,592 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,593 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:26,593 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:26,593 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:26,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d17e3748c1a841069429b0097a9cb120, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 17:57:26,595 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/aaaa6fee0e144465a7eef4e0fd4cdd30 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/aaaa6fee0e144465a7eef4e0fd4cdd30 2023-04-06 17:57:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:26,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@827731[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:26,597 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:26,597 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:26,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:26,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:26,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:26,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:26,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:26,610 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:26,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:26,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:26,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:26,610 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:26,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/aaaa6fee0e144465a7eef4e0fd4cdd30, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 17:57:26,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=128.92 KB/132012 for 9e540ddc475e6cca2398e6140b9c8e05 in 1086ms, sequenceid=13, compaction requested=false 2023-04-06 17:57:26,623 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-06 17:57:26,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:26,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,627 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:26,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:26,628 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:26,628 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:26,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:26,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,630 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:26,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:26,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=135.63 KB heapSize=388.23 KB 2023-04-06 17:57:26,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:26,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:26,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:26,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:26,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:26,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:26,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803906698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803906699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803906699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803906710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803906715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/332e1689b2b04508ac68b273c3f877f3 2023-04-06 17:57:26,728 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:26,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:26,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:26,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:26,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:26,738 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:26,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:26,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:26,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:26,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:26,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:26,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:26,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:26,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:26,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:26,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:26,758 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,758 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:26,758 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:26,758 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:26,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:26,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,759 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:26,759 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:26,762 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,763 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,763 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:26,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e7513a6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:26,764 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:26,764 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,765 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,765 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:26,767 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:26,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:26,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:26,769 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,770 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:26,770 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,770 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:26,778 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:26,778 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:26,778 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:26,778 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:26,778 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:26,785 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,789 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:26,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:26,789 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:26,789 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,791 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:26,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803906800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803906801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,889 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:26,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:26,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:26,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:26,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:26,894 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:26,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:26,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:26,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,895 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:26,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:26,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:26,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:26,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:26,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:26,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:26,899 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:26,899 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:26,899 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,899 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:26,899 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:26,899 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:26,899 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:26,900 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:26,900 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,900 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:26,900 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:26,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,901 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,901 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,901 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:26,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a583f25[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:26,903 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:26,903 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,904 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,904 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,904 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,904 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:26,904 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:26,904 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:26,905 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:26,905 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,905 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,905 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:26,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:26,908 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:26,908 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:26,908 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:26,908 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:26,908 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:26,909 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,909 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,910 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:26,911 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:26,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:26,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:26,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:26,917 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:26,917 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:26,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:26,917 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:26,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803907010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803907012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,017 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:27,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:27,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:27,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:27,021 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:27,021 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,021 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:27,021 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:27,021 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,021 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,022 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,022 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,022 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:27,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:27,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:27,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:27,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:27,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:27,023 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,023 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:27,024 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:27,024 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:27,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:27,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:27,024 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:27,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,026 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,026 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,026 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9e6c65[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:27,029 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:27,029 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,034 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,034 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:27,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:27,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:27,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:27,041 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:27,045 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:27,045 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:27,045 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:27,045 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:27,045 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:27,048 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,058 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,059 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,061 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,061 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,061 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,061 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,061 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,062 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,062 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,062 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,062 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:27,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:27,062 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,062 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:27,062 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:27,163 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:27,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:27,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:27,165 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:27,165 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,165 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:27,165 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:27,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,167 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,167 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,167 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,167 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:27,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:27,168 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:27,169 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,169 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:27,169 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:27,169 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:27,170 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,170 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,170 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:27,172 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:27,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/1e5a9718af0e4ae0a0fa698de8a43456 2023-04-06 17:57:27,172 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c36b758[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:27,172 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,172 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:27,172 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:27,172 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:27,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:27,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:27,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:27,177 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:27,177 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:27,177 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:27,177 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:27,178 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:27,178 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,179 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,179 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,179 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,180 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,181 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,181 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:27,181 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:27,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:27,181 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,182 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,182 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:27,182 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:27,182 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,182 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,182 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,182 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,282 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:27,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:27,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:27,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:27,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:27,302 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:27,302 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,302 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:27,302 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:27,302 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,302 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,303 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:27,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:27,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:27,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:27,305 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:27,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:27,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:27,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:27,314 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:27,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803907317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803907317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,325 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:27,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a68715b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:27,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:27,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:27,333 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:27,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:27,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:27,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:27,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:27,334 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:27,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:27,336 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:27,336 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:27,336 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:27,336 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:27,336 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:27,342 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,343 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:27,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:27,355 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:27,356 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:27,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,456 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:27,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:27,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:27,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:27,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:27,466 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:27,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:27,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:27,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,485 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:27,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:27,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:27,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:27,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:27,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:27,493 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:27,493 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,493 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:27,493 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:27,494 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:27,494 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:27,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:27,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:27,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:27,499 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,499 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6172dc24[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:27,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:27,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,508 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:27,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:27,510 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:27,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:27,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,511 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:27,512 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,512 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:27,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:27,514 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:27,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:27,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:27,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:27,515 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:27,516 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,516 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,516 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:27,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:27,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,519 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:27,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:27,524 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:27,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:27,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:27,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,525 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:27,525 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,525 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:27,525 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:27,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/1f99cca1745b42838c86258d1dd21cc3 2023-04-06 17:57:27,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/332e1689b2b04508ac68b273c3f877f3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/332e1689b2b04508ac68b273c3f877f3 2023-04-06 17:57:27,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/332e1689b2b04508ac68b273c3f877f3, entries=200, sequenceid=50, filesize=13.9 K 2023-04-06 17:57:27,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/1e5a9718af0e4ae0a0fa698de8a43456 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1e5a9718af0e4ae0a0fa698de8a43456 2023-04-06 17:57:27,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1e5a9718af0e4ae0a0fa698de8a43456, entries=150, sequenceid=50, filesize=11.6 K 2023-04-06 17:57:27,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/1f99cca1745b42838c86258d1dd21cc3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f99cca1745b42838c86258d1dd21cc3 2023-04-06 17:57:27,625 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:27,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:27,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f99cca1745b42838c86258d1dd21cc3, entries=150, sequenceid=50, filesize=11.6 K 2023-04-06 17:57:27,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~135.63 KB/138882, heapSize ~388.19 KB/397504, currentSize=53.67 KB/54960 for 9e540ddc475e6cca2398e6140b9c8e05 in 953ms, sequenceid=50, compaction requested=false 2023-04-06 17:57:27,627 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:27,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:27,639 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:27,639 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,639 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:27,639 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:27,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:27,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:27,641 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,641 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:27,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:27,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:27,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:27,642 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:27,642 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:27,643 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:57:27,643 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:27,643 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:27,643 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:27,643 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:27,643 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:27,643 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:27,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:27,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:27,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803907745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803847725, max timestamp=1680803847745Num uniques 859; 2023-04-06 17:57:27,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:27,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803847725, max timestamp=1680803847745Num uniques 859; 2023-04-06 17:57:27,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803847725, max timestamp=1680803847745Num uniques 859; 2023-04-06 17:57:27,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:27,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:57:27,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803907778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803907778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803907820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803907820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803907855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803907883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:27,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:27,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803907885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,059 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/fae61a16a5a643c2be128ab99ef7193e 2023-04-06 17:57:28,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803908063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803908087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803908094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803908370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803908392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803908399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,501 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/8202c5de29da4b8da3a1f4bb5a099568 2023-04-06 17:57:28,518 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/6abf64169f76436a9e1621545b031c73 2023-04-06 17:57:28,527 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/fae61a16a5a643c2be128ab99ef7193e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fae61a16a5a643c2be128ab99ef7193e 2023-04-06 17:57:28,537 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fae61a16a5a643c2be128ab99ef7193e, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 17:57:28,538 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/8202c5de29da4b8da3a1f4bb5a099568 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/8202c5de29da4b8da3a1f4bb5a099568 2023-04-06 17:57:28,550 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/8202c5de29da4b8da3a1f4bb5a099568, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 17:57:28,551 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/6abf64169f76436a9e1621545b031c73 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6abf64169f76436a9e1621545b031c73 2023-04-06 17:57:28,564 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6abf64169f76436a9e1621545b031c73, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 17:57:28,566 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9e540ddc475e6cca2398e6140b9c8e05 in 923ms, sequenceid=61, compaction requested=true 2023-04-06 17:57:28,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:28,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:28,567 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:28,567 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:28,567 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,567 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:28,567 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:28,581 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 17:57:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:28,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:28,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:28,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d83692b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:28,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:28,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:28,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:28,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:28,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:28,633 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:28,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:28,633 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:28,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:28,639 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:28,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:28,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:28,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:28,640 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:28,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:28,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:28,641 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:28,642 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:28,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:28,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:28,652 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,652 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:28,652 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:28,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:28,652 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:28,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:28,652 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:28,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:28,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:28,654 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:28,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:28,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:28,754 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:28,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:28,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:28,759 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:28,759 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:28,759 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:28,759 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:28,759 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,759 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:28,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,760 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:28,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:28,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:28,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:28,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:28,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:28,761 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:28,762 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:28,762 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-06 17:57:28,762 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:28,762 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:28,762 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:28,762 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:28,762 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:28,762 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:28,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:28,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:28,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803908845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803908847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803908873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803908899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803908905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803908948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:28,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:28,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803908951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803909155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803909159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,188 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/59ff181393664b299e7b46421e66661b 2023-04-06 17:57:29,246 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/c4024bdc5f754fd4a3ca05d935f0a89b 2023-04-06 17:57:29,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803909459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803909464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,693 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/9e452c77694d4325be3f7d4932b4852a 2023-04-06 17:57:29,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/59ff181393664b299e7b46421e66661b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/59ff181393664b299e7b46421e66661b 2023-04-06 17:57:29,718 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/59ff181393664b299e7b46421e66661b, entries=150, sequenceid=96, filesize=11.6 K 2023-04-06 17:57:29,719 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/c4024bdc5f754fd4a3ca05d935f0a89b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c4024bdc5f754fd4a3ca05d935f0a89b 2023-04-06 17:57:29,778 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c4024bdc5f754fd4a3ca05d935f0a89b, entries=150, sequenceid=96, filesize=11.6 K 2023-04-06 17:57:29,778 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/9e452c77694d4325be3f7d4932b4852a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/9e452c77694d4325be3f7d4932b4852a 2023-04-06 17:57:29,790 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/9e452c77694d4325be3f7d4932b4852a, entries=150, sequenceid=96, filesize=11.6 K 2023-04-06 17:57:29,792 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9e540ddc475e6cca2398e6140b9c8e05 in 1029ms, sequenceid=96, compaction requested=true 2023-04-06 17:57:29,792 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:29,792 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:29,795 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,795 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:29,796 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,796 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:29,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:29,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d69fb4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:29,798 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:29,799 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,799 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:29,799 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:29,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:29,800 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,800 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:29,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:29,800 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:29,801 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:29,800 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:29,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:29,804 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:29,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:29,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:29,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:29,804 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:29,805 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,805 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:29,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:29,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:29,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:29,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:29,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:29,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:29,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:29,818 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:29,818 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:29,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:57:29,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:29,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:29,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:29,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:29,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:29,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:29,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/1e1547dca0144d60bf311002bfcc4536 2023-04-06 17:57:29,918 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:29,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:29,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:29,922 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:29,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:29,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:29,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:29,923 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,923 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:29,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:29,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:29,924 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:29,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,924 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,924 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:29,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:29,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:29,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803909923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803909923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:29,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:29,929 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:29,929 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:29,929 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:29,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,930 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:29,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803909928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,930 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:29,930 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:29,930 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:29,932 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,932 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:29,933 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,933 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:29,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:29,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dadc2dc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:29,934 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:29,934 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,949 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:29,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:29,949 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:29,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:29,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:29,953 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:29,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:29,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:29,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:29,953 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:29,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:29,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:29,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803909964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:29,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803909969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,971 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:29,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:29,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:29,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:29,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:29,973 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:29,973 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:29,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:29,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:30,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803910029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803910029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803910036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,073 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:30,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:30,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:30,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:30,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:30,082 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:30,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:30,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:30,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:30,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:30,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:30,087 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,087 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,087 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:30,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:30,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:30,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:30,091 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:30,091 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:30,091 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:30,093 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:30,093 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:30,093 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,093 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:30,094 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:30,094 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,107 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:30,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ac9a8b1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:30,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:30,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,109 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:30,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:30,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:30,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:30,113 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:30,113 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:30,113 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:30,113 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:30,113 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:30,114 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,114 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:30,114 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:30,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:30,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:30,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:30,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:30,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:30,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:30,127 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:30,128 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:30,229 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:30,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:30,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:30,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803910238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803910238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:30,242 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:30,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:30,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:30,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:30,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:30,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:30,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:30,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803910244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:30,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:30,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:30,248 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:30,248 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:30,248 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:30,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:30,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:30,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:30,258 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,259 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,259 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:30,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7548f2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:30,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:30,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,261 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:30,261 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,261 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:30,262 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:30,262 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,262 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,262 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:30,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:30,265 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:30,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:30,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:30,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:30,265 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:30,266 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,266 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,266 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:30,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,270 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:30,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:30,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,277 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:30,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:30,277 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:30,277 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:30,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:30,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/c70722f61a72428288571d41dfd2c160 2023-04-06 17:57:30,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/644ede10c35744879754e8cb4b59f7de 2023-04-06 17:57:30,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/1e1547dca0144d60bf311002bfcc4536 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1e1547dca0144d60bf311002bfcc4536 2023-04-06 17:57:30,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1e1547dca0144d60bf311002bfcc4536, entries=200, sequenceid=108, filesize=13.9 K 2023-04-06 17:57:30,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/c70722f61a72428288571d41dfd2c160 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c70722f61a72428288571d41dfd2c160 2023-04-06 17:57:30,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c70722f61a72428288571d41dfd2c160, entries=150, sequenceid=108, filesize=11.6 K 2023-04-06 17:57:30,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/644ede10c35744879754e8cb4b59f7de as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/644ede10c35744879754e8cb4b59f7de 2023-04-06 17:57:30,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/644ede10c35744879754e8cb4b59f7de, entries=150, sequenceid=108, filesize=11.6 K 2023-04-06 17:57:30,377 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:30,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:30,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9e540ddc475e6cca2398e6140b9c8e05 in 491ms, sequenceid=108, compaction requested=true 2023-04-06 17:57:30,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:30,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:30,378 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:57:30,378 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:57:30,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:30,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:30,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:30,379 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64110 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:57:30,379 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:57:30,379 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/A is initiating minor compaction (all files) 2023-04-06 17:57:30,380 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/A in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,379 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/B is initiating minor compaction (all files) 2023-04-06 17:57:30,380 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/B in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,380 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ae9c6ceb8494db28933a1e319d8c25f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/332e1689b2b04508ac68b273c3f877f3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fae61a16a5a643c2be128ab99ef7193e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/59ff181393664b299e7b46421e66661b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1e1547dca0144d60bf311002bfcc4536] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=62.6 K 2023-04-06 17:57:30,380 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d17e3748c1a841069429b0097a9cb120, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1e5a9718af0e4ae0a0fa698de8a43456, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/8202c5de29da4b8da3a1f4bb5a099568, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c4024bdc5f754fd4a3ca05d935f0a89b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c70722f61a72428288571d41dfd2c160] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=58.0 K 2023-04-06 17:57:30,380 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 1ae9c6ceb8494db28933a1e319d8c25f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680803845531 2023-04-06 17:57:30,380 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d17e3748c1a841069429b0097a9cb120, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680803845531 2023-04-06 17:57:30,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:30,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:30,380 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 332e1689b2b04508ac68b273c3f877f3, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680803845575 2023-04-06 17:57:30,380 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:30,380 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:30,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:30,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:30,381 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 1e5a9718af0e4ae0a0fa698de8a43456, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680803845579 2023-04-06 17:57:30,381 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting fae61a16a5a643c2be128ab99ef7193e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803846687 2023-04-06 17:57:30,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,381 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 8202c5de29da4b8da3a1f4bb5a099568, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803846687 2023-04-06 17:57:30,381 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 59ff181393664b299e7b46421e66661b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680803847758 2023-04-06 17:57:30,381 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c4024bdc5f754fd4a3ca05d935f0a89b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680803847758 2023-04-06 17:57:30,381 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 1e1547dca0144d60bf311002bfcc4536, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680803848837 2023-04-06 17:57:30,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:30,382 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c70722f61a72428288571d41dfd2c160, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680803848837 2023-04-06 17:57:30,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:30,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:30,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:30,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:30,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:30,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:30,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:30,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:30,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:30,385 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:57:30,385 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:30,385 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:30,385 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:30,385 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:30,385 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:30,385 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:30,392 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#B#compaction#219 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:30,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,395 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#A#compaction#220 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:30,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,396 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:30,432 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/c1a1a72a0369495498f9abc40b84fc88 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/c1a1a72a0369495498f9abc40b84fc88 2023-04-06 17:57:30,435 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/15983ac500214ac783854c3a50a685a5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/15983ac500214ac783854c3a50a685a5 2023-04-06 17:57:30,437 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/A of 9e540ddc475e6cca2398e6140b9c8e05 into c1a1a72a0369495498f9abc40b84fc88(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:30,437 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:30,437 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/A, priority=11, startTime=1680803850378; duration=0sec 2023-04-06 17:57:30,437 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:30,438 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:57:30,440 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:57:30,440 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/C is initiating minor compaction (all files) 2023-04-06 17:57:30,440 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/C in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,440 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/aaaa6fee0e144465a7eef4e0fd4cdd30, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f99cca1745b42838c86258d1dd21cc3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6abf64169f76436a9e1621545b031c73, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/9e452c77694d4325be3f7d4932b4852a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/644ede10c35744879754e8cb4b59f7de] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=58.0 K 2023-04-06 17:57:30,440 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting aaaa6fee0e144465a7eef4e0fd4cdd30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680803845531 2023-04-06 17:57:30,441 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 1f99cca1745b42838c86258d1dd21cc3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680803845579 2023-04-06 17:57:30,441 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/B of 9e540ddc475e6cca2398e6140b9c8e05 into 15983ac500214ac783854c3a50a685a5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:30,441 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:30,441 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/B, priority=11, startTime=1680803850378; duration=0sec 2023-04-06 17:57:30,441 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:30,441 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6abf64169f76436a9e1621545b031c73, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680803846687 2023-04-06 17:57:30,442 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/42d43dc4123c4782902f0582a95210a2 2023-04-06 17:57:30,442 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9e452c77694d4325be3f7d4932b4852a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680803847758 2023-04-06 17:57:30,442 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 644ede10c35744879754e8cb4b59f7de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680803848837 2023-04-06 17:57:30,485 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#C#compaction#223 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:30,490 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/450dde3fc52f4555a9ecd747398c42a6 2023-04-06 17:57:30,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:30,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:30,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803910554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803910555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803910559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803910659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803910660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803910667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803910862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803910865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:30,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803910870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,916 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/dc1db5d861a24a099c0e315026b6e92b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/dc1db5d861a24a099c0e315026b6e92b 2023-04-06 17:57:30,922 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/01c281e9d04348fcaa131f45ca4e909c 2023-04-06 17:57:30,924 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/C of 9e540ddc475e6cca2398e6140b9c8e05 into dc1db5d861a24a099c0e315026b6e92b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:30,924 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:30,924 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/C, priority=11, startTime=1680803850378; duration=0sec 2023-04-06 17:57:30,924 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:30,927 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/42d43dc4123c4782902f0582a95210a2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/42d43dc4123c4782902f0582a95210a2 2023-04-06 17:57:30,932 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/42d43dc4123c4782902f0582a95210a2, entries=150, sequenceid=128, filesize=11.6 K 2023-04-06 17:57:30,933 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/450dde3fc52f4555a9ecd747398c42a6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/450dde3fc52f4555a9ecd747398c42a6 2023-04-06 17:57:30,936 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/450dde3fc52f4555a9ecd747398c42a6, entries=150, sequenceid=128, filesize=11.6 K 2023-04-06 17:57:30,937 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/01c281e9d04348fcaa131f45ca4e909c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/01c281e9d04348fcaa131f45ca4e909c 2023-04-06 17:57:30,941 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/01c281e9d04348fcaa131f45ca4e909c, entries=150, sequenceid=128, filesize=11.6 K 2023-04-06 17:57:30,941 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9e540ddc475e6cca2398e6140b9c8e05 in 556ms, sequenceid=128, compaction requested=false 2023-04-06 17:57:30,941 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:30,941 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:30,941 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:30,941 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:30,941 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,941 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:30,942 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:30,947 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:30,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5e49f7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:30,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:30,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,958 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:30,958 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:30,958 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:30,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:30,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:30,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:30,959 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:30,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:30,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:30,970 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:30,970 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:30,970 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:30,970 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:30,970 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:30,975 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,975 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:30,975 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:30,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:30,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:30,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:30,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:30,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:30,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:57:30,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:30,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:30,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:30,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:30,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:30,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:30,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:30,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:30,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:30,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:30,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:30,988 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:30,988 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:30,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:30,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:30,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:30,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:31,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/7df1ff5b59a64be991503c0dbeac67fd 2023-04-06 17:57:31,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803911013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803911013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,088 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:31,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:31,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:31,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:31,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:31,098 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:31,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:31,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:31,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,099 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:31,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:31,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,099 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:31,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:31,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:31,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:31,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:31,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:31,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:31,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:31,106 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,106 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:31,106 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:31,106 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:31,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:31,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:31,107 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:31,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:31,109 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:31,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa6c582[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:31,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:31,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:31,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:31,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:31,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:31,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:31,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803911117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803911117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:31,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:31,119 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:31,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:31,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:31,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:31,119 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:31,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:31,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:31,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:31,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:31,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:31,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:31,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:31,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:31,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:31,126 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:31,126 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:31,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803911169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803911174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803911180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,226 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:31,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:31,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:31,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:31,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:31,231 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:31,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:31,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:31,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:31,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:31,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:31,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:31,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:31,238 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:31,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:31,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:31,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:31,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:31,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:31,243 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:31,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:31,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:31,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:31,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:31,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:31,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78bd74e7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:31,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:31,250 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:31,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:31,258 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:31,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:31,260 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:31,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:31,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:31,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:31,260 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:31,261 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,261 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,261 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,261 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,262 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:31,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:31,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:31,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:31,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:31,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:31,264 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:31,264 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:31,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:31,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803911320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803911321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,364 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:31,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:31,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:31,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:31,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:31,368 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:31,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:31,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:31,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,370 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:31,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:31,370 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:31,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:31,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:31,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:31,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:31,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:31,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:31,376 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,376 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:31,376 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:31,376 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:31,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:31,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:31,377 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:31,377 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,378 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,378 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:31,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:31,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8991da[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:31,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:31,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,392 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,392 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:31,392 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:31,392 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:31,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:31,393 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,393 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:31,393 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,393 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:31,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:31,406 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:31,406 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:31,406 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:31,406 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:31,406 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:31,410 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,410 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:31,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,410 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:31,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,411 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,412 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:31,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:31,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:31,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:31,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:31,416 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:31,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:31,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:31,416 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:31,416 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:31,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/bf624b96333f46438ee6b7a30c86ab7f 2023-04-06 17:57:31,465 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/2cc9ee31ac074d56887c6924591c4f90 2023-04-06 17:57:31,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/7df1ff5b59a64be991503c0dbeac67fd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/7df1ff5b59a64be991503c0dbeac67fd 2023-04-06 17:57:31,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/7df1ff5b59a64be991503c0dbeac67fd, entries=150, sequenceid=145, filesize=11.8 K 2023-04-06 17:57:31,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/bf624b96333f46438ee6b7a30c86ab7f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bf624b96333f46438ee6b7a30c86ab7f 2023-04-06 17:57:31,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bf624b96333f46438ee6b7a30c86ab7f, entries=150, sequenceid=145, filesize=11.8 K 2023-04-06 17:57:31,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/2cc9ee31ac074d56887c6924591c4f90 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/2cc9ee31ac074d56887c6924591c4f90 2023-04-06 17:57:31,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/2cc9ee31ac074d56887c6924591c4f90, entries=150, sequenceid=145, filesize=11.8 K 2023-04-06 17:57:31,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9e540ddc475e6cca2398e6140b9c8e05 in 519ms, sequenceid=145, compaction requested=true 2023-04-06 17:57:31,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:31,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:31,498 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:31,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:31,498 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:31,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:31,499 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35980 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:31,499 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35980 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:31,499 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/A is initiating minor compaction (all files) 2023-04-06 17:57:31,499 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/B is initiating minor compaction (all files) 2023-04-06 17:57:31,499 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/A in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,499 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/B in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,499 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/15983ac500214ac783854c3a50a685a5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/450dde3fc52f4555a9ecd747398c42a6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bf624b96333f46438ee6b7a30c86ab7f] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=35.1 K 2023-04-06 17:57:31,499 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/c1a1a72a0369495498f9abc40b84fc88, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/42d43dc4123c4782902f0582a95210a2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/7df1ff5b59a64be991503c0dbeac67fd] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=35.1 K 2023-04-06 17:57:31,499 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 15983ac500214ac783854c3a50a685a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680803848837 2023-04-06 17:57:31,499 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c1a1a72a0369495498f9abc40b84fc88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680803848837 2023-04-06 17:57:31,499 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 450dde3fc52f4555a9ecd747398c42a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680803849915 2023-04-06 17:57:31,500 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 42d43dc4123c4782902f0582a95210a2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680803849915 2023-04-06 17:57:31,500 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting bf624b96333f46438ee6b7a30c86ab7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680803850553 2023-04-06 17:57:31,500 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 7df1ff5b59a64be991503c0dbeac67fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680803850553 2023-04-06 17:57:31,516 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:31,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:31,523 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#B#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:31,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:31,547 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#A#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:31,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:31,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:31,548 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:31,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:31,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:31,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,553 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:31,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,554 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:31,557 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,557 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:31,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:31,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:31,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:31,566 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:31,566 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:31,567 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:31,569 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:31,569 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:31,569 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,569 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:31,570 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:57:31,570 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:31,570 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:31,570 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:31,570 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:31,570 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:31,570 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:31,613 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/5c5d19f48d82486e8a114c66874d2734 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/5c5d19f48d82486e8a114c66874d2734 2023-04-06 17:57:31,626 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/A of 9e540ddc475e6cca2398e6140b9c8e05 into 5c5d19f48d82486e8a114c66874d2734(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:31,626 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:31,626 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/A, priority=13, startTime=1680803851497; duration=0sec 2023-04-06 17:57:31,626 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:31,626 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:31,627 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35980 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:31,627 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/C is initiating minor compaction (all files) 2023-04-06 17:57:31,627 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/C in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:31,628 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/dc1db5d861a24a099c0e315026b6e92b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/01c281e9d04348fcaa131f45ca4e909c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/2cc9ee31ac074d56887c6924591c4f90] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=35.1 K 2023-04-06 17:57:31,628 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting dc1db5d861a24a099c0e315026b6e92b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680803848837 2023-04-06 17:57:31,629 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 01c281e9d04348fcaa131f45ca4e909c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680803849915 2023-04-06 17:57:31,630 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 2cc9ee31ac074d56887c6924591c4f90, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680803850553 2023-04-06 17:57:31,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:31,631 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:31,641 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#C#compaction#231 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:31,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803911661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803911662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803911675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803911681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803911682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,692 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/ae45cb3034004e97aa332874e47331e2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/ae45cb3034004e97aa332874e47331e2 2023-04-06 17:57:31,696 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/C of 9e540ddc475e6cca2398e6140b9c8e05 into ae45cb3034004e97aa332874e47331e2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:31,697 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:31,697 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/C, priority=13, startTime=1680803851498; duration=0sec 2023-04-06 17:57:31,697 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:31,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803911771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803911772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,980 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/bed47ae9e4664d6db1c5e0c6cd4d73e7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bed47ae9e4664d6db1c5e0c6cd4d73e7 2023-04-06 17:57:31,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803911979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:31,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803911980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:31,985 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/B of 9e540ddc475e6cca2398e6140b9c8e05 into bed47ae9e4664d6db1c5e0c6cd4d73e7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:31,985 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:31,985 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/B, priority=13, startTime=1680803851498; duration=0sec 2023-04-06 17:57:31,985 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:32,026 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/2d42697c8d374c25912ca2810842b4da 2023-04-06 17:57:32,056 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/4fdc4189a454480bbb0923c7f6b32046 2023-04-06 17:57:32,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803912282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803912286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,495 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/4323e8e6e6794ea5b2cdb10a7c362cfc 2023-04-06 17:57:32,498 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/2d42697c8d374c25912ca2810842b4da as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/2d42697c8d374c25912ca2810842b4da 2023-04-06 17:57:32,501 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/2d42697c8d374c25912ca2810842b4da, entries=150, sequenceid=163, filesize=11.8 K 2023-04-06 17:57:32,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/4fdc4189a454480bbb0923c7f6b32046 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/4fdc4189a454480bbb0923c7f6b32046 2023-04-06 17:57:32,505 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/4fdc4189a454480bbb0923c7f6b32046, entries=150, sequenceid=163, filesize=11.8 K 2023-04-06 17:57:32,506 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/4323e8e6e6794ea5b2cdb10a7c362cfc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4323e8e6e6794ea5b2cdb10a7c362cfc 2023-04-06 17:57:32,511 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4323e8e6e6794ea5b2cdb10a7c362cfc, entries=150, sequenceid=163, filesize=11.8 K 2023-04-06 17:57:32,512 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 9e540ddc475e6cca2398e6140b9c8e05 in 943ms, sequenceid=163, compaction requested=false 2023-04-06 17:57:32,512 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:32,512 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:32,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:32,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:32,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:32,512 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:32,513 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:32,513 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,513 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:32,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:32,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c67e0d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:32,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:32,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,515 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,515 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:32,515 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:32,515 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:32,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:32,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:32,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:32,519 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:32,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:32,522 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:32,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:32,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:32,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:32,522 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:32,532 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:32,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:32,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:32,533 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:32,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,540 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:32,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:32,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:32,540 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:32,540 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:32,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:32,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:32,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:32,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:32,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:32,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:32,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:32,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,641 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:32,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:32,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:32,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:32,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:32,645 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:32,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:32,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:32,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:32,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:32,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:32,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:32,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:32,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:32,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:32,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:32,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:32,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:32,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:32,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:32,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:32,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:32,647 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:32,647 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:32,648 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 17:57:32,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:32,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:32,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:32,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:32,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:32,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:32,666 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/74e3c59c10a24846b2d5d3b0319a9ef9 2023-04-06 17:57:32,685 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42 2023-04-06 17:57:32,688 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:32,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803912712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803912714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803912715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803912787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803912789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803912815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803912818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:32,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:32,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803912826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803913018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803913022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803913034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,112 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/1f1a716fdf8e4995be5eace3fbafdf37 2023-04-06 17:57:33,116 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/74e3c59c10a24846b2d5d3b0319a9ef9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/74e3c59c10a24846b2d5d3b0319a9ef9 2023-04-06 17:57:33,119 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/74e3c59c10a24846b2d5d3b0319a9ef9, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 17:57:33,120 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42 2023-04-06 17:57:33,123 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 17:57:33,123 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/1f1a716fdf8e4995be5eace3fbafdf37 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f1a716fdf8e4995be5eace3fbafdf37 2023-04-06 17:57:33,128 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f1a716fdf8e4995be5eace3fbafdf37, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 17:57:33,128 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 9e540ddc475e6cca2398e6140b9c8e05 in 480ms, sequenceid=181, compaction requested=true 2023-04-06 17:57:33,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:33,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:33,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:33,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:33,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:33,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:33,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:33,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:33,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:33,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:33,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79952542[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:33,132 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:33,132 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,133 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,133 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:33,133 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:33,133 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:33,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:33,134 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:33,134 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:33,134 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,134 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:33,136 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:33,136 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:33,136 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:33,136 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:33,136 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:33,137 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,137 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:33,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,137 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:33,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:33,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:33,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:33,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:33,140 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,141 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,141 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:33,141 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:33,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:33,141 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:33,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:33,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:33,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:33,141 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:33,141 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:33,142 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:33,141 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:33,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:33,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:33,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,242 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:33,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:33,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:33,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:33,244 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:33,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:33,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:33,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:33,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:33,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,245 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:33,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:33,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:33,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:33,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:33,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:33,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:33,253 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:33,253 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:33,253 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:33,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:33,254 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:33,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:33,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:33,254 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:57:33,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:33,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:33,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:33,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:33,254 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:33,255 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:33,275 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/a90cc87e74ce4b938d06645d57e20d2e 2023-04-06 17:57:33,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:33,333 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:33,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803913401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803913406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803913410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803913509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803913510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803913518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,697 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/f1fedecc0a6e46b782ba60a5fff598ad 2023-04-06 17:57:33,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803913711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803913720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803913724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:33,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803913797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:33,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803913797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803914037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803914037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803914038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,137 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/6e53ac5eacc84245bc026e5cbd99585d 2023-04-06 17:57:34,146 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/a90cc87e74ce4b938d06645d57e20d2e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/a90cc87e74ce4b938d06645d57e20d2e 2023-04-06 17:57:34,161 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/a90cc87e74ce4b938d06645d57e20d2e, entries=150, sequenceid=199, filesize=11.8 K 2023-04-06 17:57:34,162 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/f1fedecc0a6e46b782ba60a5fff598ad as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f1fedecc0a6e46b782ba60a5fff598ad 2023-04-06 17:57:34,166 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f1fedecc0a6e46b782ba60a5fff598ad, entries=150, sequenceid=199, filesize=11.8 K 2023-04-06 17:57:34,168 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/6e53ac5eacc84245bc026e5cbd99585d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6e53ac5eacc84245bc026e5cbd99585d 2023-04-06 17:57:34,173 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6e53ac5eacc84245bc026e5cbd99585d, entries=150, sequenceid=199, filesize=11.8 K 2023-04-06 17:57:34,174 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 9e540ddc475e6cca2398e6140b9c8e05 in 920ms, sequenceid=199, compaction requested=true 2023-04-06 17:57:34,174 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:34,174 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:34,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:34,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:34,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:34,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:34,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,178 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:34,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:34,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1c6cdb[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:34,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:34,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,191 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:34,191 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:34,191 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:34,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:34,192 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,192 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:34,192 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:34,192 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:34,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:34,200 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:34,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:34,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:34,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:34,200 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:34,207 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,208 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:34,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,208 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:34,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:34,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,210 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:34,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:34,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:34,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:34,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:34,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:34,217 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:34,217 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:34,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:34,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,318 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:34,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:34,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:34,324 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:34,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:34,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:34,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:34,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:34,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:34,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:34,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:34,327 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,328 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,328 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:34,330 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:34,330 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:34,330 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:34,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:34,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:34,334 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:34,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:34,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:34,546 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:34,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803914569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803914569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803914569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803914674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803914675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803914689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,787 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/1b99864bac724c469b9e3baa276dfef2 2023-04-06 17:57:34,816 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/f2f837f644934de2b924a0a7224bf164 2023-04-06 17:57:34,836 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/39de84c5833b4778b6f1b0ecfefaec23 2023-04-06 17:57:34,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/1b99864bac724c469b9e3baa276dfef2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1b99864bac724c469b9e3baa276dfef2 2023-04-06 17:57:34,844 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1b99864bac724c469b9e3baa276dfef2, entries=150, sequenceid=214, filesize=11.8 K 2023-04-06 17:57:34,845 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/f2f837f644934de2b924a0a7224bf164 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f2f837f644934de2b924a0a7224bf164 2023-04-06 17:57:34,859 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f2f837f644934de2b924a0a7224bf164, entries=150, sequenceid=214, filesize=11.8 K 2023-04-06 17:57:34,860 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/39de84c5833b4778b6f1b0ecfefaec23 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/39de84c5833b4778b6f1b0ecfefaec23 2023-04-06 17:57:34,862 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/39de84c5833b4778b6f1b0ecfefaec23, entries=150, sequenceid=214, filesize=11.8 K 2023-04-06 17:57:34,863 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9e540ddc475e6cca2398e6140b9c8e05 in 529ms, sequenceid=214, compaction requested=true 2023-04-06 17:57:34,863 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:34,863 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:34,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:34,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:34,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:34,863 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:34,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:34,865 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,865 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:34,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:34,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58292703[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:34,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:34,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:34,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:34,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:34,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:34,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:34,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:34,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:34,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:34,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:34,877 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:34,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:34,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:34,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:34,877 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:34,883 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,883 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:34,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,885 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:34,887 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:34,887 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:34,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:34,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:34,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:34,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:34,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:34,887 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:34,887 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:34,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:34,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:34,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:34,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:34,891 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:57:34,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:34,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:34,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:34,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:34,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:34,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:34,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803914930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803914934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:34,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803914937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,988 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:34,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:34,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:34,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:34,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:34,991 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:34,991 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:34,991 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:34,991 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:34,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,992 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:34,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:34,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:34,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:34,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:35,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:35,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:35,002 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:35,002 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,002 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:35,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:35,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:35,004 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,004 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:35,004 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:35,004 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:35,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:35,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:35,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:35,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:35,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b0c0c1d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:35,027 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:35,027 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:35,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:35,029 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:35,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:35,032 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:35,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:35,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:35,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:35,032 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:35,035 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,035 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,035 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,041 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803915037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803915040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803915041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,043 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,049 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:35,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:35,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:35,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,049 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,050 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:35,050 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:35,150 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:35,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:35,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:35,161 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:35,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:35,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:35,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,166 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,167 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,167 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:35,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:35,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:35,175 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:35,175 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,175 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:35,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:35,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:35,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:35,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:35,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:35,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:35,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1be38f0b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:35,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:35,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,207 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:35,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:35,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:35,211 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:35,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:35,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:35,211 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:35,211 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:35,212 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,212 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,212 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,214 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:35,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:35,215 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,216 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,216 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,216 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,216 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:35,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,215 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,216 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:35,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:35,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803915245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803915245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803915246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,316 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:35,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:35,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:35,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/f43cfdea70b04d4eaaa00db9bd5d94a6 2023-04-06 17:57:35,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:35,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:35,334 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:35,334 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,334 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:35,334 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:35,335 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,335 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,336 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,336 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,337 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,337 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:35,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:35,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:35,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:35,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:35,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:35,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:35,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:35,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:35,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12104218[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:35,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:35,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:35,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:35,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:35,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:35,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:35,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:35,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:35,366 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:35,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:35,376 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:35,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:35,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:35,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:35,376 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:35,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:35,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:35,382 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:35,382 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:35,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:35,482 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:35,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:35,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:35,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:35,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:35,490 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:35,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:35,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:35,493 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,494 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:35,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:35,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:35,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:35,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:35,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:35,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:35,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:35,500 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,500 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:35,500 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:35,500 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:35,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:35,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:35,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:35,516 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,516 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,517 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:35,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3107e66[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:35,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:35,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,523 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:35,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:35,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:35,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:35,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:35,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:35,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:35,536 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:35,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:35,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:35,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:35,536 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:35,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:35,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:35,549 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:35,549 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:35,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803915549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803915551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803915552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:35,649 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:35,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:35,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:35,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:35,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:35,654 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:35,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:35,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:35,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,656 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:35,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:35,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:35,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:35,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:35,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:35,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:35,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:35,663 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,663 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:35,663 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:35,663 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:35,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:35,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:35,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:35,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:35,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2857e6fd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:35,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:35,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,677 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,678 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:35,678 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:35,678 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:35,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:35,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,679 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:35,679 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,679 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:35,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:35,681 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:35,681 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:35,681 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:35,681 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:35,681 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:35,682 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,682 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,683 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,684 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,687 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:35,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:35,687 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,687 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:35,687 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:35,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:35,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/5e2e3e9c577140379ca7ab713e3fd3da 2023-04-06 17:57:35,788 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:35,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:35,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:35,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:35,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:35,798 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:35,798 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,799 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:35,799 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:35,799 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,799 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,800 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,800 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:35,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:35,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:35,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:35,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:35,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:35,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:35,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:35,808 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,808 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:35,808 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:35,808 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:35,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:35,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:35,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:35,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803915809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,815 DEBUG [Thread-1651] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:35,818 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,818 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,818 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,818 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:35,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fcbbf2b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:35,820 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:35,821 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,824 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:35,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:35,824 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:35,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803915820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,825 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:35,825 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:35,826 DEBUG [Thread-1653] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:35,826 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,826 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:35,826 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,826 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:35,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:35,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/4274e15821ef4bec9cb76e06cff95377 2023-04-06 17:57:35,829 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:35,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:35,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:35,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:35,829 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:35,831 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,831 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:35,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,834 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:35,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:35,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:35,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:35,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:35,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:35,844 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:35,844 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:35,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/f43cfdea70b04d4eaaa00db9bd5d94a6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f43cfdea70b04d4eaaa00db9bd5d94a6 2023-04-06 17:57:35,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f43cfdea70b04d4eaaa00db9bd5d94a6, entries=150, sequenceid=233, filesize=11.8 K 2023-04-06 17:57:35,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/5e2e3e9c577140379ca7ab713e3fd3da as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/5e2e3e9c577140379ca7ab713e3fd3da 2023-04-06 17:57:35,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/5e2e3e9c577140379ca7ab713e3fd3da, entries=150, sequenceid=233, filesize=11.8 K 2023-04-06 17:57:35,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/4274e15821ef4bec9cb76e06cff95377 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4274e15821ef4bec9cb76e06cff95377 2023-04-06 17:57:35,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4274e15821ef4bec9cb76e06cff95377, entries=150, sequenceid=233, filesize=11.8 K 2023-04-06 17:57:35,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 9e540ddc475e6cca2398e6140b9c8e05 in 993ms, sequenceid=233, compaction requested=true 2023-04-06 17:57:35,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:35,885 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:35,885 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:57:35,885 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-06 17:57:35,885 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:57:35,886 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72490 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:57:35,886 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/A is initiating minor compaction (all files) 2023-04-06 17:57:35,886 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/A in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,886 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/5c5d19f48d82486e8a114c66874d2734, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/2d42697c8d374c25912ca2810842b4da, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/74e3c59c10a24846b2d5d3b0319a9ef9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/a90cc87e74ce4b938d06645d57e20d2e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1b99864bac724c469b9e3baa276dfef2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f43cfdea70b04d4eaaa00db9bd5d94a6] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=70.8 K 2023-04-06 17:57:35,887 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5c5d19f48d82486e8a114c66874d2734, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680803850553 2023-04-06 17:57:35,887 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2d42697c8d374c25912ca2810842b4da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680803851008 2023-04-06 17:57:35,887 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 74e3c59c10a24846b2d5d3b0319a9ef9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803851641 2023-04-06 17:57:35,887 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a90cc87e74ce4b938d06645d57e20d2e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680803852706 2023-04-06 17:57:35,888 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 1b99864bac724c469b9e3baa276dfef2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680803853335 2023-04-06 17:57:35,888 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f43cfdea70b04d4eaaa00db9bd5d94a6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803854567 2023-04-06 17:57:35,889 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:57:35,905 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72490 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:57:35,905 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/C is initiating minor compaction (all files) 2023-04-06 17:57:35,905 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/C in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,905 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/ae45cb3034004e97aa332874e47331e2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4323e8e6e6794ea5b2cdb10a7c362cfc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f1a716fdf8e4995be5eace3fbafdf37, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6e53ac5eacc84245bc026e5cbd99585d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/39de84c5833b4778b6f1b0ecfefaec23, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4274e15821ef4bec9cb76e06cff95377] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=70.8 K 2023-04-06 17:57:35,908 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ae45cb3034004e97aa332874e47331e2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680803850553 2023-04-06 17:57:35,908 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4323e8e6e6794ea5b2cdb10a7c362cfc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680803851008 2023-04-06 17:57:35,909 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 1f1a716fdf8e4995be5eace3fbafdf37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803851641 2023-04-06 17:57:35,909 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6e53ac5eacc84245bc026e5cbd99585d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680803852706 2023-04-06 17:57:35,909 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 39de84c5833b4778b6f1b0ecfefaec23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680803853335 2023-04-06 17:57:35,910 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 4274e15821ef4bec9cb76e06cff95377, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803854567 2023-04-06 17:57:35,917 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#A#compaction#246 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:35,921 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#C#compaction#247 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:35,951 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:35,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:35,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:35,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:35,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:35,958 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:35,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:35,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:35,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:35,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:35,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:35,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:35,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:35,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:35,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:35,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:35,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:35,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:35,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:35,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:35,973 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:35,973 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:35,974 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 17:57:35,974 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:35,974 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:35,974 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:35,974 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:35,974 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:35,974 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:36,015 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/6274c56df14240b4b537ae58d2522df4 2023-04-06 17:57:36,046 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/b0f4adaa34cb421faa3e820a7c403cb6 2023-04-06 17:57:36,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:36,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:36,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803916080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47042 deadline: 1680803916083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47018 deadline: 1680803916083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,089 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/50f9bc2762b441188fbc619992eb9011 2023-04-06 17:57:36,093 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/6274c56df14240b4b537ae58d2522df4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/6274c56df14240b4b537ae58d2522df4 2023-04-06 17:57:36,103 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/6274c56df14240b4b537ae58d2522df4, entries=150, sequenceid=247, filesize=11.8 K 2023-04-06 17:57:36,105 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/b0f4adaa34cb421faa3e820a7c403cb6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/b0f4adaa34cb421faa3e820a7c403cb6 2023-04-06 17:57:36,108 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/b0f4adaa34cb421faa3e820a7c403cb6, entries=150, sequenceid=247, filesize=11.8 K 2023-04-06 17:57:36,109 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/50f9bc2762b441188fbc619992eb9011 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/50f9bc2762b441188fbc619992eb9011 2023-04-06 17:57:36,115 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/50f9bc2762b441188fbc619992eb9011, entries=150, sequenceid=247, filesize=11.8 K 2023-04-06 17:57:36,116 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9e540ddc475e6cca2398e6140b9c8e05 in 142ms, sequenceid=247, compaction requested=true 2023-04-06 17:57:36,116 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:36,116 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:36,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:36,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:36,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:36,117 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:36,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fdb2848[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:36,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:36,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:36,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:36,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:36,140 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,140 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:36,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:36,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:36,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:36,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:36,145 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:36,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:36,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:36,145 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:36,145 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:36,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,158 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,164 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,173 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:36,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:36,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,173 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:36,173 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:36,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:36,174 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,174 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:36,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:57:36,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:36,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:36,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:36,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:36,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:36,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:36,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/028a9d58f2c74965b08d384a841e5409 2023-04-06 17:57:36,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803916254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803916255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803916258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,275 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:36,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:36,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:36,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:36,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:36,278 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:36,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:36,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:36,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:36,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:36,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:36,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:36,281 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:36,281 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,281 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:36,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:36,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:36,282 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,282 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:36,282 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:36,282 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:36,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:36,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:36,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:36,284 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:36,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f38929b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:36,286 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:36,286 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,286 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,287 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:36,287 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:36,287 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:36,287 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,287 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:36,288 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:36,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:36,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:36,290 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:36,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:36,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:36,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:36,291 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:36,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,292 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,294 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,295 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:36,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,310 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:36,310 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,361 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/1ac6fb92901d4275850294457831049e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ac6fb92901d4275850294457831049e 2023-04-06 17:57:36,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803916363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803916365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803916365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,370 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/A of 9e540ddc475e6cca2398e6140b9c8e05 into 1ac6fb92901d4275850294457831049e(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:36,370 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:36,370 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/A, priority=10, startTime=1680803855885; duration=0sec 2023-04-06 17:57:36,370 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:36,370 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 17:57:36,378 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84526 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 17:57:36,378 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/B is initiating minor compaction (all files) 2023-04-06 17:57:36,378 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/B in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,378 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bed47ae9e4664d6db1c5e0c6cd4d73e7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/4fdc4189a454480bbb0923c7f6b32046, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f1fedecc0a6e46b782ba60a5fff598ad, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f2f837f644934de2b924a0a7224bf164, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/5e2e3e9c577140379ca7ab713e3fd3da, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/b0f4adaa34cb421faa3e820a7c403cb6] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=82.5 K 2023-04-06 17:57:36,379 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting bed47ae9e4664d6db1c5e0c6cd4d73e7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680803850553 2023-04-06 17:57:36,379 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4fdc4189a454480bbb0923c7f6b32046, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680803851008 2023-04-06 17:57:36,380 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7cd7f9cbc9c949ac8bed6bb9d6ac1c42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803851641 2023-04-06 17:57:36,380 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f1fedecc0a6e46b782ba60a5fff598ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680803852706 2023-04-06 17:57:36,380 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f2f837f644934de2b924a0a7224bf164, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680803853335 2023-04-06 17:57:36,381 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5e2e3e9c577140379ca7ab713e3fd3da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803854567 2023-04-06 17:57:36,381 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b0f4adaa34cb421faa3e820a7c403cb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680803854928 2023-04-06 17:57:36,385 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/429757045efe487f9085b5cef4b45952 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/429757045efe487f9085b5cef4b45952 2023-04-06 17:57:36,389 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/C of 9e540ddc475e6cca2398e6140b9c8e05 into 429757045efe487f9085b5cef4b45952(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:36,390 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:36,390 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/C, priority=10, startTime=1680803855885; duration=0sec 2023-04-06 17:57:36,390 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:36,399 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#B#compaction#253 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:36,410 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:36,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:36,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:36,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:36,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:36,426 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:36,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:36,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:36,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,427 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,428 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:36,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:36,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:36,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:36,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:36,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:36,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:36,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:36,434 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:36,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:36,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,442 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:36,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@284feb8f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:36,449 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:36,449 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,449 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:36,449 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:36,449 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:36,449 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,449 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:36,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:36,450 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:36,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:36,452 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:36,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:36,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:36,452 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:36,452 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:36,453 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,453 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,453 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,454 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,455 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,456 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,456 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,457 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,457 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:36,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:36,457 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:36,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,457 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,458 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,458 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,458 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,458 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,457 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:36,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:36,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,558 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:36,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:36,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:36,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:36,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:36,561 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:36,561 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,561 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:36,561 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:36,561 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,561 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,562 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,562 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,562 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,562 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:36,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:36,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:36,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:36,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:36,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:36,563 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:36,565 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,565 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:36,565 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:36,565 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,565 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:36,565 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:36,565 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,565 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:36,565 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:36,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803916568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803916568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803916568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,569 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,569 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,570 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,570 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:36,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8f2b4a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:36,572 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:36,572 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,574 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,574 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:36,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:36,575 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:36,575 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:36,575 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:36,575 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,575 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,575 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:36,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:36,578 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:36,578 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:36,578 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:36,578 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:36,578 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:36,580 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,581 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,581 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,581 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,582 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,583 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:36,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:36,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,584 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,584 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,584 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:36,584 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,584 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:36,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:36,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/1658d8d69e62461bb95ebc3e6f4850a3 2023-04-06 17:57:36,684 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:36,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:36,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:36,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:36,690 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:36,690 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,690 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:36,690 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:36,690 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,690 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,691 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,691 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,691 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:36,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:36,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:36,692 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:36,693 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,693 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:36,696 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:36,696 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:36,696 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,696 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:36,696 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:36,696 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,696 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:36,696 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:36,696 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,696 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:36,696 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:36,699 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,700 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,700 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:36,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c70d13b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:36,701 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:36,702 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,702 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:36,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:36,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:36,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,703 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:36,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:36,709 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:36,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:36,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:36,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:36,709 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:36,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,717 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:36,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:36,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:36,724 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:36,724 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:36,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,725 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:36,824 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:36,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:36,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:36,831 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/970c9a2f3e2d4190a315a70e9a885d50 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/970c9a2f3e2d4190a315a70e9a885d50 2023-04-06 17:57:36,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:36,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:36,833 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:36,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:36,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:36,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:36,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:36,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:36,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:36,835 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,835 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:36,835 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:36,835 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:36,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:36,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47a73064[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:36,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:36,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:36,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:36,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:36,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:36,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:36,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:36,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,842 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/B of 9e540ddc475e6cca2398e6140b9c8e05 into 970c9a2f3e2d4190a315a70e9a885d50(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:36,842 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:36,842 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/B, priority=9, startTime=1680803855885; duration=0sec 2023-04-06 17:57:36,842 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:36,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:36,857 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:36,857 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:36,857 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:36,857 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:36,857 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:36,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,860 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,861 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:36,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,866 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:36,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:36,866 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:36,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:36,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803916870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803916871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:36,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803916871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,967 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:36,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:36,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:36,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:36,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:36,971 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:36,971 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,971 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:36,971 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:36,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:36,977 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:36,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:36,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:36,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:36,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:36,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:36,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:36,978 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:36,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:36,979 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:36,981 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,981 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:36,981 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:36,981 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:36,990 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:36,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:36,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:36,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:36,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1063b88c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:36,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:36,999 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:36,999 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:36,999 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:37,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:37,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:37,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:37,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:37,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:37,014 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:37,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:37,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:37,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:37,014 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:37,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:37,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,020 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:37,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:37,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:37,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:37,025 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:37,025 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:37,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:37,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:37,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:37,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:37,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/6853a518d69f403aaa35292b4ced1d22 2023-04-06 17:57:37,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/028a9d58f2c74965b08d384a841e5409 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/028a9d58f2c74965b08d384a841e5409 2023-04-06 17:57:37,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/028a9d58f2c74965b08d384a841e5409, entries=200, sequenceid=267, filesize=14.2 K 2023-04-06 17:57:37,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/1658d8d69e62461bb95ebc3e6f4850a3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1658d8d69e62461bb95ebc3e6f4850a3 2023-04-06 17:57:37,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1658d8d69e62461bb95ebc3e6f4850a3, entries=150, sequenceid=267, filesize=11.9 K 2023-04-06 17:57:37,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/6853a518d69f403aaa35292b4ced1d22 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6853a518d69f403aaa35292b4ced1d22 2023-04-06 17:57:37,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6853a518d69f403aaa35292b4ced1d22, entries=150, sequenceid=267, filesize=11.9 K 2023-04-06 17:57:37,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for 9e540ddc475e6cca2398e6140b9c8e05 in 878ms, sequenceid=267, compaction requested=true 2023-04-06 17:57:37,071 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:37,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:37,071 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:37,071 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 17:57:37,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:37,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:37,072 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:37,072 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:37,072 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. because compaction request was cancelled 2023-04-06 17:57:37,072 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:37,072 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39126 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:37,072 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/A is initiating minor compaction (all files) 2023-04-06 17:57:37,072 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/A in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:37,072 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ac6fb92901d4275850294457831049e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/6274c56df14240b4b537ae58d2522df4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/028a9d58f2c74965b08d384a841e5409] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=38.2 K 2023-04-06 17:57:37,073 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 1ac6fb92901d4275850294457831049e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803854567 2023-04-06 17:57:37,073 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6274c56df14240b4b537ae58d2522df4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680803854928 2023-04-06 17:57:37,073 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36686 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:37,073 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/C is initiating minor compaction (all files) 2023-04-06 17:57:37,073 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/C in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:37,073 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/429757045efe487f9085b5cef4b45952, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/50f9bc2762b441188fbc619992eb9011, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6853a518d69f403aaa35292b4ced1d22] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=35.8 K 2023-04-06 17:57:37,073 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 429757045efe487f9085b5cef4b45952, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803854567 2023-04-06 17:57:37,074 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 50f9bc2762b441188fbc619992eb9011, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680803854928 2023-04-06 17:57:37,074 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 028a9d58f2c74965b08d384a841e5409, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680803856068 2023-04-06 17:57:37,074 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6853a518d69f403aaa35292b4ced1d22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680803856068 2023-04-06 17:57:37,096 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#C#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:37,103 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#A#compaction#256 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:37,125 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:37,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:37,136 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/048ce6d9eb654e54a51f16cb8d19318f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/048ce6d9eb654e54a51f16cb8d19318f 2023-04-06 17:57:37,143 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/C of 9e540ddc475e6cca2398e6140b9c8e05 into 048ce6d9eb654e54a51f16cb8d19318f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:37,143 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:37,143 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/C, priority=13, startTime=1680803857071; duration=0sec 2023-04-06 17:57:37,143 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:37,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:37,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:37,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:37,158 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:37,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:37,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:37,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:37,164 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,164 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:37,165 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/fd8acaa520bd4b889d99ba68f8e4337b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fd8acaa520bd4b889d99ba68f8e4337b 2023-04-06 17:57:37,170 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,170 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,170 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:37,175 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:37,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:37,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:37,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:37,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:37,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:37,186 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:37,186 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:37,186 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 17:57:37,186 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:37,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:37,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:37,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:37,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:37,187 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:37,189 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/A of 9e540ddc475e6cca2398e6140b9c8e05 into fd8acaa520bd4b889d99ba68f8e4337b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:37,189 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:37,189 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/A, priority=13, startTime=1680803857071; duration=0sec 2023-04-06 17:57:37,189 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:37,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:37,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803917401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803917402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803917403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803917506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803917506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803917522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,611 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/08e524bf8b0f47f091cce5f56eadf6af 2023-04-06 17:57:37,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803917712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803917712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:37,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:37,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803917724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803918014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803918016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47078 deadline: 1680803918028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,042 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/54fb001a18b644f28e99d76ba66f2c41 2023-04-06 17:57:38,450 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/15f88d53876a465e9118b1df277d3a9e 2023-04-06 17:57:38,465 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/08e524bf8b0f47f091cce5f56eadf6af as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/08e524bf8b0f47f091cce5f56eadf6af 2023-04-06 17:57:38,470 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/08e524bf8b0f47f091cce5f56eadf6af, entries=150, sequenceid=285, filesize=11.9 K 2023-04-06 17:57:38,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/54fb001a18b644f28e99d76ba66f2c41 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/54fb001a18b644f28e99d76ba66f2c41 2023-04-06 17:57:38,479 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/54fb001a18b644f28e99d76ba66f2c41, entries=150, sequenceid=285, filesize=11.9 K 2023-04-06 17:57:38,479 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/15f88d53876a465e9118b1df277d3a9e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/15f88d53876a465e9118b1df277d3a9e 2023-04-06 17:57:38,493 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/15f88d53876a465e9118b1df277d3a9e, entries=150, sequenceid=285, filesize=11.9 K 2023-04-06 17:57:38,494 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9e540ddc475e6cca2398e6140b9c8e05 in 1308ms, sequenceid=285, compaction requested=true 2023-04-06 17:57:38,494 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:38,494 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:38,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:38,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:38,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:38,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:38,495 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:38,506 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339504b1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:38,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:38,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:38,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:57:38,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:38,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:38,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:38,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:38,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:38,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:38,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,542 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:38,542 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:38,542 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:38,543 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:38,543 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:38,543 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:38,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:38,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:38,544 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:38,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:38,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:38,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:38,544 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:38,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:38,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:38,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:38,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:38,554 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:38,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:38,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:38,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:38,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,555 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:38,555 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:38,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:38,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803918556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803918557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803918562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/43ece17fb6524d6286588a433ae5877d 2023-04-06 17:57:38,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/d38cd94c234f40c78d5768d1c86283da 2023-04-06 17:57:38,655 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:38,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:38,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803918663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803918663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803918668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:38,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:38,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:38,678 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:38,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:38,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:38,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:38,684 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,684 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,685 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:38,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:38,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:38,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:38,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:38,688 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:38,688 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:38,688 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:38,688 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:38,688 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:38,689 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:38,689 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:38,689 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:38,689 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:38,689 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:38,689 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:38,689 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,689 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:38,689 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:38,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,698 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:38,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:38,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d6e2f1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:38,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,700 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,700 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:38,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:38,700 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:38,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:38,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:38,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,701 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:38,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:38,703 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:38,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:38,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:38,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:38,703 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:38,704 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,704 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,704 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,705 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:38,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:38,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:38,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:38,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,709 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:38,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:38,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:38,709 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:38,810 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:38,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:38,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:38,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:38,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:38,814 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:38,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:38,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:38,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:38,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,815 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:38,815 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,816 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,816 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:38,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:38,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:38,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:38,819 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:38,819 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:38,819 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:38,822 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:38,822 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:38,822 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:38,822 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:38,822 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:38,822 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:38,822 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:38,822 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:38,822 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,822 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:38,822 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:38,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:38,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@778cd7bf[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:38,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:38,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:38,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:38,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:38,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:38,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:38,849 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:38,849 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:38,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:38,852 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:38,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:38,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:38,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:38,853 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:38,858 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,858 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:38,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,859 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,861 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:38,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:38,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:38,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:38,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:38,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:38,866 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:38,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:38,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,866 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:38,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:38,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803918866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803918867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:38,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803918870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,967 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:38,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:38,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:38,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:38,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:38,972 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:38,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:38,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:38,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:38,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:38,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:38,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:38,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:38,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:38,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:38,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:38,986 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:38,986 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:38,986 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:38,986 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:38,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:38,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:38,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aa6a61a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:38,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:38,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:38,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:38,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:38,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:38,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:38,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:38,996 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:38,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:38,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:38,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:38,996 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:38,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:38,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:38,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:38,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,003 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,004 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:39,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:39,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/bbe3e870ded847fa803030ecbf111037 2023-04-06 17:57:39,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/43ece17fb6524d6286588a433ae5877d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/43ece17fb6524d6286588a433ae5877d 2023-04-06 17:57:39,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/43ece17fb6524d6286588a433ae5877d, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 17:57:39,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/d38cd94c234f40c78d5768d1c86283da as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d38cd94c234f40c78d5768d1c86283da 2023-04-06 17:57:39,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d38cd94c234f40c78d5768d1c86283da, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 17:57:39,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/bbe3e870ded847fa803030ecbf111037 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/bbe3e870ded847fa803030ecbf111037 2023-04-06 17:57:39,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/bbe3e870ded847fa803030ecbf111037, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 17:57:39,052 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9e540ddc475e6cca2398e6140b9c8e05 in 516ms, sequenceid=306, compaction requested=true 2023-04-06 17:57:39,052 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:39,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:39,052 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:39,052 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:39,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:39,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:39,053 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:39,053 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/A is initiating minor compaction (all files) 2023-04-06 17:57:39,053 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/A in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,053 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fd8acaa520bd4b889d99ba68f8e4337b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/08e524bf8b0f47f091cce5f56eadf6af, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/43ece17fb6524d6286588a433ae5877d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=36.2 K 2023-04-06 17:57:39,053 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49056 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:39,053 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/B is initiating minor compaction (all files) 2023-04-06 17:57:39,053 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/B in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,053 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting fd8acaa520bd4b889d99ba68f8e4337b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680803856068 2023-04-06 17:57:39,053 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/970c9a2f3e2d4190a315a70e9a885d50, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1658d8d69e62461bb95ebc3e6f4850a3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/54fb001a18b644f28e99d76ba66f2c41, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d38cd94c234f40c78d5768d1c86283da] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=47.9 K 2023-04-06 17:57:39,054 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 08e524bf8b0f47f091cce5f56eadf6af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680803856210 2023-04-06 17:57:39,054 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 970c9a2f3e2d4190a315a70e9a885d50, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680803854928 2023-04-06 17:57:39,054 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 43ece17fb6524d6286588a433ae5877d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803857397 2023-04-06 17:57:39,054 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 1658d8d69e62461bb95ebc3e6f4850a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680803856068 2023-04-06 17:57:39,054 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 54fb001a18b644f28e99d76ba66f2c41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680803856210 2023-04-06 17:57:39,054 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d38cd94c234f40c78d5768d1c86283da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803857397 2023-04-06 17:57:39,059 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#A#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:39,071 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#B#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:39,104 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:39,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:39,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:39,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:39,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:39,107 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:39,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:39,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:39,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:39,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:39,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:39,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:39,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:39,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:39,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:39,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:39,109 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:39,110 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:57:39,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:39,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:39,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:39,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:39,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:39,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:39,126 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/8f6d48066b4d4290bc11f26565aadaac 2023-04-06 17:57:39,143 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/6b31eda630b5400792e330b089b73db1 2023-04-06 17:57:39,156 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/43caa8fd6ab24951bacc8c6310fa56f0 2023-04-06 17:57:39,160 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/8f6d48066b4d4290bc11f26565aadaac as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/8f6d48066b4d4290bc11f26565aadaac 2023-04-06 17:57:39,164 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/8f6d48066b4d4290bc11f26565aadaac, entries=150, sequenceid=318, filesize=11.9 K 2023-04-06 17:57:39,164 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/6b31eda630b5400792e330b089b73db1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6b31eda630b5400792e330b089b73db1 2023-04-06 17:57:39,168 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6b31eda630b5400792e330b089b73db1, entries=150, sequenceid=318, filesize=11.9 K 2023-04-06 17:57:39,169 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/43caa8fd6ab24951bacc8c6310fa56f0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/43caa8fd6ab24951bacc8c6310fa56f0 2023-04-06 17:57:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:39,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:39,173 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/43caa8fd6ab24951bacc8c6310fa56f0, entries=150, sequenceid=318, filesize=11.9 K 2023-04-06 17:57:39,174 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=13.42 KB/13740 for 9e540ddc475e6cca2398e6140b9c8e05 in 64ms, sequenceid=318, compaction requested=true 2023-04-06 17:57:39,174 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:39,174 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:39,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:39,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:39,174 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:39,175 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,175 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:39,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:39,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610d8573[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:39,176 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:39,176 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,177 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:39,178 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,178 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:39,178 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:39,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:39,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:57:39,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:39,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:39,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:39,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:39,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:39,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:39,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:39,192 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:39,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:39,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:39,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:39,192 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:39,193 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,193 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,193 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,198 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/0174cf20a46b4fa58d9665283ca46fcd 2023-04-06 17:57:39,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,206 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,207 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,207 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,207 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:39,207 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803859203, max timestamp=1680803859213Num uniques 859; 2023-04-06 17:57:39,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:57:39,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803859203, max timestamp=1680803859213Num uniques 859; 2023-04-06 17:57:39,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:57:39,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680803859203, max timestamp=1680803859213Num uniques 859; 2023-04-06 17:57:39,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 17:57:39,222 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/3b7e4a7a1050418a825cafe4bbb4c54c 2023-04-06 17:57:39,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/3ec1e73a240a45f793b9ffca2a855ca4 2023-04-06 17:57:39,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/0174cf20a46b4fa58d9665283ca46fcd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/0174cf20a46b4fa58d9665283ca46fcd 2023-04-06 17:57:39,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/0174cf20a46b4fa58d9665283ca46fcd, entries=200, sequenceid=330, filesize=14.3 K 2023-04-06 17:57:39,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/3b7e4a7a1050418a825cafe4bbb4c54c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3b7e4a7a1050418a825cafe4bbb4c54c 2023-04-06 17:57:39,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3b7e4a7a1050418a825cafe4bbb4c54c, entries=150, sequenceid=330, filesize=11.9 K 2023-04-06 17:57:39,241 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/3ec1e73a240a45f793b9ffca2a855ca4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/3ec1e73a240a45f793b9ffca2a855ca4 2023-04-06 17:57:39,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/3ec1e73a240a45f793b9ffca2a855ca4, entries=150, sequenceid=330, filesize=11.9 K 2023-04-06 17:57:39,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=112.87 KB/115578 for 9e540ddc475e6cca2398e6140b9c8e05 in 63ms, sequenceid=330, compaction requested=true 2023-04-06 17:57:39,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:39,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:57:39,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-06 17:57:39,246 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-06 17:57:39,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:39,246 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=119.58 KB heapSize=342.40 KB 2023-04-06 17:57:39,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:39,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:39,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:39,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:39,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:39,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:39,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803919258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803919261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803919262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,307 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:39,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:39,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:39,310 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:39,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:39,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:39,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,311 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,311 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,311 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,311 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:39,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:39,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:39,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:39,312 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,312 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:39,312 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:39,312 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:39,313 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f7232c9[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:39,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:39,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,315 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:39,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:39,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:39,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:39,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:39,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:39,319 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:39,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:39,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:39,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:39,319 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:39,320 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,320 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,320 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,320 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,322 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,325 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,325 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:39,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803919363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803919366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803919366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,425 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:39,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:39,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:39,428 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:39,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:39,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:39,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,429 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:39,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:39,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:39,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:39,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:39,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:39,430 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:39,430 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,430 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:39,431 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:39,431 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:39,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:39,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:39,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e5227b7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:39,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:39,434 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:39,435 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:39,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:39,437 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:39,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:39,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:39,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:39,437 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:39,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,439 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,440 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:39,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,441 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,441 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:39,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,479 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/568b998d99db45f4b25297641df7eef7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/568b998d99db45f4b25297641df7eef7 2023-04-06 17:57:39,483 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/A of 9e540ddc475e6cca2398e6140b9c8e05 into 568b998d99db45f4b25297641df7eef7(size=12.6 K), total size for store is 38.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:39,483 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:39,483 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/A, priority=13, startTime=1680803859052; duration=0sec 2023-04-06 17:57:39,483 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-06 17:57:39,484 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-06 17:57:39,484 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:39,484 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:39,484 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. because compaction request was cancelled 2023-04-06 17:57:39,484 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:57:39,485 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/2e361ed2ced0402b8a1dd8e857778c7f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2e361ed2ced0402b8a1dd8e857778c7f 2023-04-06 17:57:39,485 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61460 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:57:39,486 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/C is initiating minor compaction (all files) 2023-04-06 17:57:39,486 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/C in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,486 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/048ce6d9eb654e54a51f16cb8d19318f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/15f88d53876a465e9118b1df277d3a9e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/bbe3e870ded847fa803030ecbf111037, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/43caa8fd6ab24951bacc8c6310fa56f0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/3ec1e73a240a45f793b9ffca2a855ca4] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=60.0 K 2023-04-06 17:57:39,486 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 048ce6d9eb654e54a51f16cb8d19318f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680803856068 2023-04-06 17:57:39,487 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 15f88d53876a465e9118b1df277d3a9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680803856210 2023-04-06 17:57:39,487 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting bbe3e870ded847fa803030ecbf111037, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803857397 2023-04-06 17:57:39,487 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 43caa8fd6ab24951bacc8c6310fa56f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680803858544 2023-04-06 17:57:39,487 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3ec1e73a240a45f793b9ffca2a855ca4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803859175 2023-04-06 17:57:39,489 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/B of 9e540ddc475e6cca2398e6140b9c8e05 into 2e361ed2ced0402b8a1dd8e857778c7f(size=12.5 K), total size for store is 36.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:39,489 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:39,489 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/B, priority=12, startTime=1680803859052; duration=0sec 2023-04-06 17:57:39,489 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:57:39,489 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-06 17:57:39,489 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:39,489 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:39,489 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. because compaction request was cancelled 2023-04-06 17:57:39,489 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:57:39,490 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:57:39,490 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/A is initiating minor compaction (all files) 2023-04-06 17:57:39,490 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/A in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,490 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/568b998d99db45f4b25297641df7eef7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/8f6d48066b4d4290bc11f26565aadaac, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/0174cf20a46b4fa58d9665283ca46fcd] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=38.8 K 2023-04-06 17:57:39,490 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 568b998d99db45f4b25297641df7eef7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803857397 2023-04-06 17:57:39,491 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 8f6d48066b4d4290bc11f26565aadaac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680803858544 2023-04-06 17:57:39,491 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0174cf20a46b4fa58d9665283ca46fcd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803859175 2023-04-06 17:57:39,494 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#C#compaction#272 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:39,497 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#A#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:39,541 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:39,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:39,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:39,543 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:39,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:39,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:39,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,544 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:39,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:39,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:39,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:39,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:39,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:39,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:39,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:39,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:39,547 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:39,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:39,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10bce11[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:39,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:39,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:39,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:39,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:39,550 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,550 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,550 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,550 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:39,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:39,552 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:39,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:39,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:39,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:39,552 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:39,553 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,553 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,553 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,554 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,556 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,556 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,556 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:39,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803919568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803919569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803919569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/9f430e17211544b5a0fb72ee9f4620ab 2023-04-06 17:57:39,657 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:39,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:39,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:39,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:39,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:39,660 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:39,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:39,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:39,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,662 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:39,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:39,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:39,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:39,674 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:39,674 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:39,675 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,675 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:39,675 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:39,675 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:39,676 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca0e55e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:39,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:39,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,678 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,678 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:39,678 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:39,679 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:39,679 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,679 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:39,679 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:39,679 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,679 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:39,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:39,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:39,682 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:39,682 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:39,682 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:39,682 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:39,682 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:39,683 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,683 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,684 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,684 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,687 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,687 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,687 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:39,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,787 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:39,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:39,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:39,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:39,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:39,790 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:39,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:39,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:39,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,792 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:39,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:39,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:39,793 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,793 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:39,793 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:39,794 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:39,794 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,795 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,795 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:39,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428a7a79[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:39,796 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:39,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,802 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,802 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:39,802 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:39,802 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:39,802 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,802 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:39,803 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,803 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,803 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,803 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:39,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:39,806 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:39,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:39,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:39,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:39,806 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:39,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,811 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,812 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:39,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47028 deadline: 1680803919831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,835 DEBUG [Thread-1651] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:39,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47044 deadline: 1680803919859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,862 DEBUG [Thread-1653] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:39,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803919870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803919872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:39,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803919874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,903 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/789783572af84f8aaf002b08bbab2695 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/789783572af84f8aaf002b08bbab2695 2023-04-06 17:57:39,907 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/C of 9e540ddc475e6cca2398e6140b9c8e05 into 789783572af84f8aaf002b08bbab2695(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:39,907 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:39,907 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/C, priority=11, startTime=1680803859246; duration=0sec 2023-04-06 17:57:39,907 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:39,909 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/ae81a21bddca4391af57ba2139b58bd9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ae81a21bddca4391af57ba2139b58bd9 2023-04-06 17:57:39,912 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:39,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:39,912 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/A of 9e540ddc475e6cca2398e6140b9c8e05 into ae81a21bddca4391af57ba2139b58bd9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:39,912 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:39,912 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/A, priority=13, startTime=1680803859245; duration=0sec 2023-04-06 17:57:39,913 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:39,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:39,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:39,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:39,915 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:39,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:39,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:39,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:39,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:39,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:39,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:39,917 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,918 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:39,918 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:39,918 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:39,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:39,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:39,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:39,918 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3a9200[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:39,920 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:39,920 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,921 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,921 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:39,921 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:39,921 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:39,921 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:39,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,921 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,922 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,922 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:39,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:39,924 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:39,924 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:39,924 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:39,924 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:39,924 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:39,925 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,925 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,925 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:39,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,926 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:39,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,927 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:39,928 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:39,928 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:39,928 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:39,928 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:39,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:39,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:39,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:39,929 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:39,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:39,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:39,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:39,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:39,929 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:39,929 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:40,029 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:40,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:40,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:40,032 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:40,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:40,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:40,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,033 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:40,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:40,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:40,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:40,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:40,034 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,034 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:40,035 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:40,035 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:40,035 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,036 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,036 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:40,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429eacf0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:40,037 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:40,037 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,037 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:40,038 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,038 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:40,038 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:40,040 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:40,040 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:40,040 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:40,040 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:40,040 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:40,041 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,041 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,041 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,042 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,043 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,044 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,044 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,044 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,044 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:40,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:40,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:40,044 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,044 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:40,044 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:40,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,044 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,045 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,045 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,045 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,045 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/3543610ddc3f471eb83c00109362d30f 2023-04-06 17:57:40,145 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:40,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:40,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:40,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:40,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:40,147 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:40,147 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,147 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:40,147 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:40,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,148 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:40,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:40,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:40,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:40,151 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,151 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:40,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:40,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@491f2be8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:40,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,153 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,153 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:40,153 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:40,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:40,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:40,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:40,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:40,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:40,156 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:40,156 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:40,156 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:40,156 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:40,156 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:40,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,159 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,160 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:40,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:40,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,160 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:40,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,160 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:40,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,260 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:40,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:40,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:40,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:40,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:40,263 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:40,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:40,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:40,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:40,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:40,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:40,265 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:40,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:40,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:40,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:40,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:40,267 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:40,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d7d900[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:40,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:40,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:40,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:40,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:40,269 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,269 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:40,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,270 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:40,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:40,271 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:40,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:40,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:40,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:40,271 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:40,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,272 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,273 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:40,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:40,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,275 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:40,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:40,275 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:40,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:40,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803920372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,375 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:40,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:40,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:40,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:40,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:40,378 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:40,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:40,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:40,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:40,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803920377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:40,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:40,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:40,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:40,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:40,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:40,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:40,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:40,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803920379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,381 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:40,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11b4ea88[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:40,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:40,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:40,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:40,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:40,384 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,384 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:40,384 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:40,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:40,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:40,386 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:40,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:40,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:40,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:40,386 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:40,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:40,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:40,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,390 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:40,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,390 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:40,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:40,491 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:40,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:40,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:40,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/f8a32fc58d314d2bbe6b56c247102720 2023-04-06 17:57:40,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:40,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:40,495 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:40,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:40,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:40,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,496 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,497 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:40,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:40,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:40,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:40,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:40,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/9f430e17211544b5a0fb72ee9f4620ab as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9f430e17211544b5a0fb72ee9f4620ab 2023-04-06 17:57:40,498 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,498 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:40,498 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:40,498 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:40,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:40,499 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,499 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,499 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,499 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:40,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267a2d62[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:40,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:40,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9f430e17211544b5a0fb72ee9f4620ab, entries=200, sequenceid=366, filesize=14.3 K 2023-04-06 17:57:40,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/3543610ddc3f471eb83c00109362d30f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3543610ddc3f471eb83c00109362d30f 2023-04-06 17:57:40,502 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:40,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:40,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:40,502 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,502 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:40,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:40,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,504 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:40,504 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3543610ddc3f471eb83c00109362d30f, entries=150, sequenceid=366, filesize=11.9 K 2023-04-06 17:57:40,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:40,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:40,506 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:40,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:40,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:40,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:40,506 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:40,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/f8a32fc58d314d2bbe6b56c247102720 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/f8a32fc58d314d2bbe6b56c247102720 2023-04-06 17:57:40,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,507 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/f8a32fc58d314d2bbe6b56c247102720, entries=150, sequenceid=366, filesize=11.9 K 2023-04-06 17:57:40,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~126.29 KB/129318, heapSize ~362.27 KB/370968, currentSize=53.67 KB/54960 for 9e540ddc475e6cca2398e6140b9c8e05 in 1264ms, sequenceid=366, compaction requested=true 2023-04-06 17:57:40,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:40,510 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:40,510 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 17:57:40,510 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:57:40,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:40,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:40,510 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:40,510 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. because compaction request was cancelled 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/B is initiating minor compaction (all files) 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 17:57:40,511 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/B in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. because compaction request was cancelled 2023-04-06 17:57:40,511 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2e361ed2ced0402b8a1dd8e857778c7f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6b31eda630b5400792e330b089b73db1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3b7e4a7a1050418a825cafe4bbb4c54c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3543610ddc3f471eb83c00109362d30f] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=48.2 K 2023-04-06 17:57:40,511 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2e361ed2ced0402b8a1dd8e857778c7f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680803857397 2023-04-06 17:57:40,512 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6b31eda630b5400792e330b089b73db1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680803858544 2023-04-06 17:57:40,512 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3b7e4a7a1050418a825cafe4bbb4c54c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803859175 2023-04-06 17:57:40,513 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3543610ddc3f471eb83c00109362d30f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680803859239 2023-04-06 17:57:40,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,520 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,520 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,520 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:40,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:40,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:40,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:40,520 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:40,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:40,520 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:40,521 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:40,520 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:40,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,520 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#B#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:40,621 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:40,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:40,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:40,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:40,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:40,623 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:40,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:40,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:40,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,624 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,624 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,624 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:40,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:40,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:40,625 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:40,625 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:40,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:40,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:40,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:40,625 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:40,625 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:40,625 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:40,626 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:40,626 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:40,626 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:40,626 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:40,928 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/2713b47f392641bb90ebc3cf355861a2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2713b47f392641bb90ebc3cf355861a2 2023-04-06 17:57:40,931 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/B of 9e540ddc475e6cca2398e6140b9c8e05 into 2713b47f392641bb90ebc3cf355861a2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:40,931 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:40,931 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/B, priority=12, startTime=1680803860510; duration=0sec 2023-04-06 17:57:40,931 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:41,036 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/189738f5968e4ac491bba897cc1a9cf0 2023-04-06 17:57:41,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:41,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:41,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803861394, max timestamp=1680803861398Num uniques 859; 2023-04-06 17:57:41,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:41,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803861394, max timestamp=1680803861398Num uniques 859; 2023-04-06 17:57:41,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:41,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803861394, max timestamp=1680803861398Num uniques 859; 2023-04-06 17:57:41,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:41,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803921414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803921417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803921417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,443 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/0d63a910e67d4221ae5893cd4a527275 2023-04-06 17:57:41,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803921518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803921520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803921520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803921721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803921721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:41,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803921722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,851 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/b73353cd1a8543da8df5a255cd67ec46 2023-04-06 17:57:41,855 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/189738f5968e4ac491bba897cc1a9cf0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/189738f5968e4ac491bba897cc1a9cf0 2023-04-06 17:57:41,869 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/189738f5968e4ac491bba897cc1a9cf0, entries=150, sequenceid=381, filesize=11.9 K 2023-04-06 17:57:41,870 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/0d63a910e67d4221ae5893cd4a527275 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/0d63a910e67d4221ae5893cd4a527275 2023-04-06 17:57:41,873 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/0d63a910e67d4221ae5893cd4a527275, entries=150, sequenceid=381, filesize=11.9 K 2023-04-06 17:57:41,874 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/b73353cd1a8543da8df5a255cd67ec46 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b73353cd1a8543da8df5a255cd67ec46 2023-04-06 17:57:41,877 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b73353cd1a8543da8df5a255cd67ec46, entries=150, sequenceid=381, filesize=11.9 K 2023-04-06 17:57:41,877 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 9e540ddc475e6cca2398e6140b9c8e05 in 1251ms, sequenceid=381, compaction requested=true 2023-04-06 17:57:41,877 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:41,877 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:41,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:41,880 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,880 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:41,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:41,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160a122e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:41,881 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:41,881 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:41,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:41,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:41,882 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,882 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:41,883 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:41,883 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,883 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:41,883 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:41,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:41,885 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:41,885 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:41,885 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:41,885 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:41,885 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:41,886 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,886 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:41,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,886 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:41,886 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:41,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:41,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:41,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:41,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:41,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:41,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:41,889 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,889 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:41,889 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:41,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:41,990 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:41,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:41,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:41,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:41,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:41,992 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:41,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:41,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:41,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:41,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:41,993 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:41,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:41,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:41,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:41,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:41,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:41,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:41,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:41,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:41,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:41,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:41,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:41,994 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 17:57:41,995 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:41,995 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:41,995 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:41,995 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:41,995 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:41,995 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:42,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:42,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:42,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803922030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803922031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803922032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803922133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803922134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803922136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803922334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803922337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803922338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,401 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/95834871f43143668b3d32f83108deb8 2023-04-06 17:57:42,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803922636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803922641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803922642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,810 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/3222319d13fe462c96fa9b38b2f339cf 2023-04-06 17:57:42,818 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/26e5d799e1c64be8b3f62e22e3810f13 2023-04-06 17:57:42,821 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/95834871f43143668b3d32f83108deb8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/95834871f43143668b3d32f83108deb8 2023-04-06 17:57:42,824 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/95834871f43143668b3d32f83108deb8, entries=150, sequenceid=419, filesize=11.9 K 2023-04-06 17:57:42,824 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/3222319d13fe462c96fa9b38b2f339cf as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3222319d13fe462c96fa9b38b2f339cf 2023-04-06 17:57:42,827 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3222319d13fe462c96fa9b38b2f339cf, entries=150, sequenceid=419, filesize=11.9 K 2023-04-06 17:57:42,828 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/26e5d799e1c64be8b3f62e22e3810f13 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/26e5d799e1c64be8b3f62e22e3810f13 2023-04-06 17:57:42,834 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/26e5d799e1c64be8b3f62e22e3810f13, entries=150, sequenceid=419, filesize=11.9 K 2023-04-06 17:57:42,835 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 9e540ddc475e6cca2398e6140b9c8e05 in 841ms, sequenceid=419, compaction requested=true 2023-04-06 17:57:42,835 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:42,835 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:42,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:42,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:42,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:42,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f3c636[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:42,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:42,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:42,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:42,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:42,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:42,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:42,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:42,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:42,841 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:42,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:42,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:42,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:42,841 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:42,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:42,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:42,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:42,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:42,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:42,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:42,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:42,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:42,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:42,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:42,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:42,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:42,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:42,846 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:42,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:42,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:42,846 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:42,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,947 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:42,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:42,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:42,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:42,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:42,949 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:42,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:42,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:42,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:42,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:42,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:42,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:42,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:42,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:42,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:42,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:42,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:42,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:42,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:42,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:42,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:42,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:42,952 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:42,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:42,958 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/29131343b8fd456e87b2a45efd363cc6 2023-04-06 17:57:42,965 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/a745c48669b8473ba9ce0043d20d1baf 2023-04-06 17:57:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:43,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803863146, max timestamp=1680803863185Num uniques 859; 2023-04-06 17:57:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803863146, max timestamp=1680803863185Num uniques 859; 2023-04-06 17:57:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803863146, max timestamp=1680803863185Num uniques 859; 2023-04-06 17:57:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:43,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803923224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803923225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803923227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803923329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803923329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803923332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,372 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/acc4cc7fe84e40a2a5061983c139e773 2023-04-06 17:57:43,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/29131343b8fd456e87b2a45efd363cc6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/29131343b8fd456e87b2a45efd363cc6 2023-04-06 17:57:43,378 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/29131343b8fd456e87b2a45efd363cc6, entries=150, sequenceid=429, filesize=11.9 K 2023-04-06 17:57:43,379 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/a745c48669b8473ba9ce0043d20d1baf as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/a745c48669b8473ba9ce0043d20d1baf 2023-04-06 17:57:43,382 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/a745c48669b8473ba9ce0043d20d1baf, entries=150, sequenceid=429, filesize=11.9 K 2023-04-06 17:57:43,382 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/acc4cc7fe84e40a2a5061983c139e773 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/acc4cc7fe84e40a2a5061983c139e773 2023-04-06 17:57:43,385 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/acc4cc7fe84e40a2a5061983c139e773, entries=150, sequenceid=429, filesize=11.9 K 2023-04-06 17:57:43,386 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9e540ddc475e6cca2398e6140b9c8e05 in 433ms, sequenceid=429, compaction requested=true 2023-04-06 17:57:43,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:43,386 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:43,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:43,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:43,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:43,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:43,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:43,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:43,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:43,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4625820a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:43,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:43,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,389 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:43,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:43,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:43,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:43,389 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:43,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:43,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:43,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:43,391 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:43,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:43,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:43,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:43,391 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:43,393 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,393 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:43,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,393 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:43,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:43,394 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:43,396 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:43,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:43,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:43,396 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:43,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,396 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:43,496 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:43,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:43,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:43,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:43,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:43,499 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:43,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:43,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:43,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:43,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:43,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,500 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:43,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:43,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:43,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:43,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:43,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:43,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:43,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:43,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:43,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:43,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:43,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:43,501 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:43,501 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:43,501 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 17:57:43,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:43,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:43,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:43,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:43,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:43,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:43,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:43,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:43,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803923539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803923543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803923544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803923645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803923649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803923649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803923848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803923852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803923853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:43,908 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/448d89f01bda48ccb28a86bba8de2422 2023-04-06 17:57:44,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803924152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803924155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803924156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,318 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/11e39c5275bd4200b8517c81bdd80c25 2023-04-06 17:57:44,328 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/be038a9bed554aea8e4491f9a0c631b8 2023-04-06 17:57:44,332 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/448d89f01bda48ccb28a86bba8de2422 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/448d89f01bda48ccb28a86bba8de2422 2023-04-06 17:57:44,335 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/448d89f01bda48ccb28a86bba8de2422, entries=150, sequenceid=466, filesize=11.9 K 2023-04-06 17:57:44,336 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/11e39c5275bd4200b8517c81bdd80c25 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/11e39c5275bd4200b8517c81bdd80c25 2023-04-06 17:57:44,339 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/11e39c5275bd4200b8517c81bdd80c25, entries=150, sequenceid=466, filesize=11.9 K 2023-04-06 17:57:44,340 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/be038a9bed554aea8e4491f9a0c631b8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/be038a9bed554aea8e4491f9a0c631b8 2023-04-06 17:57:44,348 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/be038a9bed554aea8e4491f9a0c631b8, entries=150, sequenceid=466, filesize=11.9 K 2023-04-06 17:57:44,348 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 9e540ddc475e6cca2398e6140b9c8e05 in 847ms, sequenceid=466, compaction requested=true 2023-04-06 17:57:44,348 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:44,348 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:44,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:44,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:44,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,353 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:44,353 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:44,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:44,355 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,355 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:44,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:44,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3143a691[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:44,356 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:44,356 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:44,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:44,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:44,357 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,357 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:44,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:44,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:44,358 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:44,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:44,360 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:44,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:44,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:44,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:44,360 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:44,361 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:44,361 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,361 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:44,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:44,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:44,362 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,363 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:44,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:44,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:44,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:44,365 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:44,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:44,366 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:44,366 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:44,466 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:44,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:44,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:44,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:44,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:44,468 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:44,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:44,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:44,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:44,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:44,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,470 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:44,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:44,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:44,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:44,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:44,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:44,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:44,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:44,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:44,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:44,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:44,471 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:44,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:44,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:44,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:44,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803864674, max timestamp=1680803864680Num uniques 859; 2023-04-06 17:57:44,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:44,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803864674, max timestamp=1680803864680Num uniques 859; 2023-04-06 17:57:44,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:44,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803864674, max timestamp=1680803864680Num uniques 859; 2023-04-06 17:57:44,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:57:44,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680803864693, max timestamp=1680803864703Num uniques 1018; 2023-04-06 17:57:44,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-06 17:57:44,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680803864693, max timestamp=1680803864703Num uniques 1018; 2023-04-06 17:57:44,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-06 17:57:44,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680803864693, max timestamp=1680803864703Num uniques 1018; 2023-04-06 17:57:44,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-06 17:57:44,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803924727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803924728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803924729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803924832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803924834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:44,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803924834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:44,877 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/09f7ff623c514553ab3794a29d7ca3ee 2023-04-06 17:57:44,885 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/bc3a474ce62840cc953ea4e304aebe9b 2023-04-06 17:57:45,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803925036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803925037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803925038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,292 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/0b8fa16a0184452cadd474d297d9bca6 2023-04-06 17:57:45,296 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/09f7ff623c514553ab3794a29d7ca3ee as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/09f7ff623c514553ab3794a29d7ca3ee 2023-04-06 17:57:45,299 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/09f7ff623c514553ab3794a29d7ca3ee, entries=150, sequenceid=476, filesize=11.9 K 2023-04-06 17:57:45,300 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/bc3a474ce62840cc953ea4e304aebe9b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bc3a474ce62840cc953ea4e304aebe9b 2023-04-06 17:57:45,303 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bc3a474ce62840cc953ea4e304aebe9b, entries=150, sequenceid=476, filesize=11.9 K 2023-04-06 17:57:45,303 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/0b8fa16a0184452cadd474d297d9bca6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0b8fa16a0184452cadd474d297d9bca6 2023-04-06 17:57:45,306 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0b8fa16a0184452cadd474d297d9bca6, entries=150, sequenceid=476, filesize=11.9 K 2023-04-06 17:57:45,307 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9e540ddc475e6cca2398e6140b9c8e05 in 836ms, sequenceid=476, compaction requested=true 2023-04-06 17:57:45,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:45,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:45,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:45,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:45,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:45,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:45,308 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,308 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:45,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:45,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:45,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@771e59f5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:45,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:45,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,310 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:45,310 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:45,310 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:45,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:45,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:45,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:45,311 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:45,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:45,313 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:45,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:45,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:45,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:45,313 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:45,314 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,314 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,314 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:45,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:45,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:45,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:45,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:45,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:45,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:45,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:45,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:45,317 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,317 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:45,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-06 17:57:45,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:45,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:45,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:45,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:45,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:45,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:57:45,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803925351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803925352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803925353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,417 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:45,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:45,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:45,420 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:45,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:45,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:45,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:45,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:45,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,421 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:45,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:45,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:45,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:45,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:45,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:45,423 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:45,423 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. started... 2023-04-06 17:57:45,423 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. as already flushing 2023-04-06 17:57:45,423 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1932f356[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:45,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:45,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,426 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:45,426 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:45,426 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:45,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:45,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:45,427 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:45,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:45,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:45,429 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:45,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:45,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:45,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:45,429 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:45,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:45,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:45,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:45,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:45,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:45,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:45,433 DEBUG [Thread-1659] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:45,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:45,433 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:45,433 DEBUG [Thread-1659] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:45,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:45,443 DEBUG [Thread-1664] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ffcf146 to 127.0.0.1:49384 2023-04-06 17:57:45,443 DEBUG [Thread-1664] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:45,444 DEBUG [Thread-1666] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fba9cd6 to 127.0.0.1:49384 2023-04-06 17:57:45,444 DEBUG [Thread-1666] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:45,445 DEBUG [Thread-1662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e9921a7 to 127.0.0.1:49384 2023-04-06 17:57:45,445 DEBUG [Thread-1662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:45,446 DEBUG [Thread-1660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24e1e878 to 127.0.0.1:49384 2023-04-06 17:57:45,446 DEBUG [Thread-1660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:45,448 DEBUG [Thread-1668] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e897a83 to 127.0.0.1:49384 2023-04-06 17:57:45,448 DEBUG [Thread-1668] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:45,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803925457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803925458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803925458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,534 DEBUG [Thread-1659] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:45,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803925659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803925660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803925660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/f94951ca71444de99d5bd688f82ff1af 2023-04-06 17:57:45,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803925962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803925964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:45,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803925965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:46,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/6541437a28c84dfc832e4cc73dbb6a2f 2023-04-06 17:57:46,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:46,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47042 deadline: 1680803926467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:46,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:46,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47018 deadline: 1680803926468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:46,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:46,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47078 deadline: 1680803926470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:46,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/b78a6186cd6f4217b9c896d492c910a9 2023-04-06 17:57:46,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/f94951ca71444de99d5bd688f82ff1af as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f94951ca71444de99d5bd688f82ff1af 2023-04-06 17:57:46,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f94951ca71444de99d5bd688f82ff1af, entries=150, sequenceid=529, filesize=11.9 K 2023-04-06 17:57:46,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/6541437a28c84dfc832e4cc73dbb6a2f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6541437a28c84dfc832e4cc73dbb6a2f 2023-04-06 17:57:46,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6541437a28c84dfc832e4cc73dbb6a2f, entries=150, sequenceid=529, filesize=11.9 K 2023-04-06 17:57:46,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/b78a6186cd6f4217b9c896d492c910a9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b78a6186cd6f4217b9c896d492c910a9 2023-04-06 17:57:46,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b78a6186cd6f4217b9c896d492c910a9, entries=150, sequenceid=529, filesize=11.9 K 2023-04-06 17:57:46,619 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for 9e540ddc475e6cca2398e6140b9c8e05 in 1276ms, sequenceid=529, compaction requested=true 2023-04-06 17:57:46,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:46,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:46,619 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 17:57:46,619 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 17:57:46,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:46,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:46,622 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86086 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 17:57:46,622 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/B is initiating minor compaction (all files) 2023-04-06 17:57:46,622 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/B in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:46,622 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 100712 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 17:57:46,622 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2713b47f392641bb90ebc3cf355861a2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/0d63a910e67d4221ae5893cd4a527275, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3222319d13fe462c96fa9b38b2f339cf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/a745c48669b8473ba9ce0043d20d1baf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/11e39c5275bd4200b8517c81bdd80c25, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bc3a474ce62840cc953ea4e304aebe9b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6541437a28c84dfc832e4cc73dbb6a2f] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=84.1 K 2023-04-06 17:57:46,622 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/A is initiating minor compaction (all files) 2023-04-06 17:57:46,623 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/A in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:46,623 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ae81a21bddca4391af57ba2139b58bd9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9f430e17211544b5a0fb72ee9f4620ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/189738f5968e4ac491bba897cc1a9cf0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/95834871f43143668b3d32f83108deb8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/29131343b8fd456e87b2a45efd363cc6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/448d89f01bda48ccb28a86bba8de2422, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/09f7ff623c514553ab3794a29d7ca3ee, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f94951ca71444de99d5bd688f82ff1af] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=98.4 K 2023-04-06 17:57:46,623 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2713b47f392641bb90ebc3cf355861a2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680803859239 2023-04-06 17:57:46,623 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ae81a21bddca4391af57ba2139b58bd9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803859175 2023-04-06 17:57:46,623 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0d63a910e67d4221ae5893cd4a527275, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680803859257 2023-04-06 17:57:46,624 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 9f430e17211544b5a0fb72ee9f4620ab, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680803859239 2023-04-06 17:57:46,624 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3222319d13fe462c96fa9b38b2f339cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680803861409 2023-04-06 17:57:46,624 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 189738f5968e4ac491bba897cc1a9cf0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680803859257 2023-04-06 17:57:46,624 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a745c48669b8473ba9ce0043d20d1baf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680803862027 2023-04-06 17:57:46,624 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 95834871f43143668b3d32f83108deb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680803861409 2023-04-06 17:57:46,625 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 11e39c5275bd4200b8517c81bdd80c25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680803863219 2023-04-06 17:57:46,625 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 29131343b8fd456e87b2a45efd363cc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680803862027 2023-04-06 17:57:46,625 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting bc3a474ce62840cc953ea4e304aebe9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680803863538 2023-04-06 17:57:46,625 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 448d89f01bda48ccb28a86bba8de2422, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680803863219 2023-04-06 17:57:46,625 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6541437a28c84dfc832e4cc73dbb6a2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680803864726 2023-04-06 17:57:46,626 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 09f7ff623c514553ab3794a29d7ca3ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680803863538 2023-04-06 17:57:46,626 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f94951ca71444de99d5bd688f82ff1af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680803864726 2023-04-06 17:57:46,638 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#B#compaction#295 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:46,638 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#A#compaction#296 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:47,054 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/12c3c554813b40e88ca603d8d9ec5007 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/12c3c554813b40e88ca603d8d9ec5007 2023-04-06 17:57:47,054 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/e93df4713a404ca4a1aebda8aa5fa4d2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/e93df4713a404ca4a1aebda8aa5fa4d2 2023-04-06 17:57:47,058 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/A of 9e540ddc475e6cca2398e6140b9c8e05 into 12c3c554813b40e88ca603d8d9ec5007(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:47,058 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/B of 9e540ddc475e6cca2398e6140b9c8e05 into e93df4713a404ca4a1aebda8aa5fa4d2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:47,058 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:47,058 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:47,058 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/A, priority=8, startTime=1680803866619; duration=0sec 2023-04-06 17:57:47,058 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/B, priority=9, startTime=1680803866619; duration=0sec 2023-04-06 17:57:47,058 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:57:47,058 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:47,058 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 17:57:47,060 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98238 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 17:57:47,060 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 9e540ddc475e6cca2398e6140b9c8e05/C is initiating minor compaction (all files) 2023-04-06 17:57:47,060 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9e540ddc475e6cca2398e6140b9c8e05/C in TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:47,060 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/789783572af84f8aaf002b08bbab2695, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/f8a32fc58d314d2bbe6b56c247102720, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b73353cd1a8543da8df5a255cd67ec46, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/26e5d799e1c64be8b3f62e22e3810f13, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/acc4cc7fe84e40a2a5061983c139e773, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/be038a9bed554aea8e4491f9a0c631b8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0b8fa16a0184452cadd474d297d9bca6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b78a6186cd6f4217b9c896d492c910a9] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp, totalSize=95.9 K 2023-04-06 17:57:47,061 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 789783572af84f8aaf002b08bbab2695, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803859175 2023-04-06 17:57:47,061 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f8a32fc58d314d2bbe6b56c247102720, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680803859239 2023-04-06 17:57:47,061 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b73353cd1a8543da8df5a255cd67ec46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680803859257 2023-04-06 17:57:47,062 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 26e5d799e1c64be8b3f62e22e3810f13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680803861409 2023-04-06 17:57:47,062 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting acc4cc7fe84e40a2a5061983c139e773, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680803862027 2023-04-06 17:57:47,062 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting be038a9bed554aea8e4491f9a0c631b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680803863219 2023-04-06 17:57:47,062 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 0b8fa16a0184452cadd474d297d9bca6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680803863538 2023-04-06 17:57:47,063 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b78a6186cd6f4217b9c896d492c910a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680803864726 2023-04-06 17:57:47,072 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9e540ddc475e6cca2398e6140b9c8e05#C#compaction#297 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:57:47,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:47,475 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:57:47,475 DEBUG [Thread-1649] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05fbdc11 to 127.0.0.1:49384 2023-04-06 17:57:47,476 DEBUG [Thread-1649] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:47,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:47,477 DEBUG [Thread-1655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76e08050 to 127.0.0.1:49384 2023-04-06 17:57:47,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:47,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:47,478 DEBUG [Thread-1655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:47,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:47,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:47,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:47,480 DEBUG [Thread-1657] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2dbba21f to 127.0.0.1:49384 2023-04-06 17:57:47,480 DEBUG [Thread-1657] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:47,484 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/710b45a53b024667a6f196ee4e9204d8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/710b45a53b024667a6f196ee4e9204d8 2023-04-06 17:57:47,488 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9e540ddc475e6cca2398e6140b9c8e05/C of 9e540ddc475e6cca2398e6140b9c8e05 into 710b45a53b024667a6f196ee4e9204d8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:57:47,488 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:47,488 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05., storeName=9e540ddc475e6cca2398e6140b9c8e05/C, priority=8, startTime=1680803866620; duration=0sec 2023-04-06 17:57:47,488 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:57:47,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/ffeb5b0e257b4391bc995ddcb8fcd253 2023-04-06 17:57:48,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/70283a2ab16a439094ebe75fcbdb32b8 2023-04-06 17:57:48,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/0db84a7d327f4c10835495d001446920 2023-04-06 17:57:48,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/ffeb5b0e257b4391bc995ddcb8fcd253 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ffeb5b0e257b4391bc995ddcb8fcd253 2023-04-06 17:57:48,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ffeb5b0e257b4391bc995ddcb8fcd253, entries=150, sequenceid=542, filesize=11.9 K 2023-04-06 17:57:48,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/70283a2ab16a439094ebe75fcbdb32b8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/70283a2ab16a439094ebe75fcbdb32b8 2023-04-06 17:57:48,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/70283a2ab16a439094ebe75fcbdb32b8, entries=150, sequenceid=542, filesize=11.9 K 2023-04-06 17:57:48,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/0db84a7d327f4c10835495d001446920 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0db84a7d327f4c10835495d001446920 2023-04-06 17:57:48,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0db84a7d327f4c10835495d001446920, entries=150, sequenceid=542, filesize=11.9 K 2023-04-06 17:57:48,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=6.71 KB/6870 for 9e540ddc475e6cca2398e6140b9c8e05 in 1287ms, sequenceid=542, compaction requested=false 2023-04-06 17:57:48,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:49,884 DEBUG [Thread-1651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6298fe55 to 127.0.0.1:49384 2023-04-06 17:57:49,885 DEBUG [Thread-1651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:49,935 DEBUG [Thread-1653] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5620345b to 127.0.0.1:49384 2023-04-06 17:57:49,935 DEBUG [Thread-1653] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:49,935 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 17:57:49,935 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 141 2023-04-06 17:57:49,935 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-06 17:57:49,935 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-06 17:57:49,935 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 4116 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 4101 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 3993 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 4205 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 4079 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 17:57:49,936 INFO [Listener at localhost.localdomain/40169] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 17:57:49,936 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a2832f4 to 127.0.0.1:49384 2023-04-06 17:57:49,936 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:57:49,937 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 17:57:49,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 17:57:49,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:49,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 17:57:49,940 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803869940"}]},"ts":"1680803869940"} 2023-04-06 17:57:49,941 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 17:57:49,942 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 17:57:49,943 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9e540ddc475e6cca2398e6140b9c8e05, UNASSIGN}] 2023-04-06 17:57:49,944 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9e540ddc475e6cca2398e6140b9c8e05, UNASSIGN 2023-04-06 17:57:49,944 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=9e540ddc475e6cca2398e6140b9c8e05, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:49,945 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803869944"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803869944"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803869944"}]},"ts":"1680803869944"} 2023-04-06 17:57:49,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:57:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 17:57:50,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 9e540ddc475e6cca2398e6140b9c8e05, disabling compactions & flushes 2023-04-06 17:57:50,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. after waiting 0 ms 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:50,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 9e540ddc475e6cca2398e6140b9c8e05 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=A 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=B 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9e540ddc475e6cca2398e6140b9c8e05, store=C 2023-04-06 17:57:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:50,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 17:57:50,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/9001c723017049cc9c2957fa9b1d5a1a 2023-04-06 17:57:50,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 17:57:50,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/f4950822a52c4ad8b50ddda05ff26cfb 2023-04-06 17:57:51,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 17:57:51,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/895c6850c1ff4b40aa7319bdad489923 2023-04-06 17:57:51,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/A/9001c723017049cc9c2957fa9b1d5a1a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9001c723017049cc9c2957fa9b1d5a1a 2023-04-06 17:57:51,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9001c723017049cc9c2957fa9b1d5a1a, entries=100, sequenceid=549, filesize=9.5 K 2023-04-06 17:57:51,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/B/f4950822a52c4ad8b50ddda05ff26cfb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f4950822a52c4ad8b50ddda05ff26cfb 2023-04-06 17:57:51,336 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f4950822a52c4ad8b50ddda05ff26cfb, entries=100, sequenceid=549, filesize=9.5 K 2023-04-06 17:57:51,336 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/.tmp/C/895c6850c1ff4b40aa7319bdad489923 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/895c6850c1ff4b40aa7319bdad489923 2023-04-06 17:57:51,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/895c6850c1ff4b40aa7319bdad489923, entries=100, sequenceid=549, filesize=9.5 K 2023-04-06 17:57:51,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 9e540ddc475e6cca2398e6140b9c8e05 in 1242ms, sequenceid=549, compaction requested=true 2023-04-06 17:57:51,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ae9c6ceb8494db28933a1e319d8c25f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/332e1689b2b04508ac68b273c3f877f3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fae61a16a5a643c2be128ab99ef7193e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/59ff181393664b299e7b46421e66661b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1e1547dca0144d60bf311002bfcc4536, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/c1a1a72a0369495498f9abc40b84fc88, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/42d43dc4123c4782902f0582a95210a2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/5c5d19f48d82486e8a114c66874d2734, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/7df1ff5b59a64be991503c0dbeac67fd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/2d42697c8d374c25912ca2810842b4da, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/74e3c59c10a24846b2d5d3b0319a9ef9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/a90cc87e74ce4b938d06645d57e20d2e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1b99864bac724c469b9e3baa276dfef2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ac6fb92901d4275850294457831049e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f43cfdea70b04d4eaaa00db9bd5d94a6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/6274c56df14240b4b537ae58d2522df4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/028a9d58f2c74965b08d384a841e5409, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fd8acaa520bd4b889d99ba68f8e4337b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/08e524bf8b0f47f091cce5f56eadf6af, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/568b998d99db45f4b25297641df7eef7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/43ece17fb6524d6286588a433ae5877d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/8f6d48066b4d4290bc11f26565aadaac, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/0174cf20a46b4fa58d9665283ca46fcd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ae81a21bddca4391af57ba2139b58bd9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9f430e17211544b5a0fb72ee9f4620ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/189738f5968e4ac491bba897cc1a9cf0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/95834871f43143668b3d32f83108deb8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/29131343b8fd456e87b2a45efd363cc6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/448d89f01bda48ccb28a86bba8de2422, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/09f7ff623c514553ab3794a29d7ca3ee, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f94951ca71444de99d5bd688f82ff1af] to archive 2023-04-06 17:57:51,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:57:51,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ae9c6ceb8494db28933a1e319d8c25f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ae9c6ceb8494db28933a1e319d8c25f 2023-04-06 17:57:51,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/332e1689b2b04508ac68b273c3f877f3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/332e1689b2b04508ac68b273c3f877f3 2023-04-06 17:57:51,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fae61a16a5a643c2be128ab99ef7193e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fae61a16a5a643c2be128ab99ef7193e 2023-04-06 17:57:51,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/59ff181393664b299e7b46421e66661b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/59ff181393664b299e7b46421e66661b 2023-04-06 17:57:51,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1e1547dca0144d60bf311002bfcc4536 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1e1547dca0144d60bf311002bfcc4536 2023-04-06 17:57:51,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/c1a1a72a0369495498f9abc40b84fc88 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/c1a1a72a0369495498f9abc40b84fc88 2023-04-06 17:57:51,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/42d43dc4123c4782902f0582a95210a2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/42d43dc4123c4782902f0582a95210a2 2023-04-06 17:57:51,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/5c5d19f48d82486e8a114c66874d2734 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/5c5d19f48d82486e8a114c66874d2734 2023-04-06 17:57:51,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/7df1ff5b59a64be991503c0dbeac67fd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/7df1ff5b59a64be991503c0dbeac67fd 2023-04-06 17:57:51,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/2d42697c8d374c25912ca2810842b4da to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/2d42697c8d374c25912ca2810842b4da 2023-04-06 17:57:51,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/74e3c59c10a24846b2d5d3b0319a9ef9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/74e3c59c10a24846b2d5d3b0319a9ef9 2023-04-06 17:57:51,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/a90cc87e74ce4b938d06645d57e20d2e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/a90cc87e74ce4b938d06645d57e20d2e 2023-04-06 17:57:51,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1b99864bac724c469b9e3baa276dfef2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1b99864bac724c469b9e3baa276dfef2 2023-04-06 17:57:51,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ac6fb92901d4275850294457831049e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/1ac6fb92901d4275850294457831049e 2023-04-06 17:57:51,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f43cfdea70b04d4eaaa00db9bd5d94a6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f43cfdea70b04d4eaaa00db9bd5d94a6 2023-04-06 17:57:51,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/6274c56df14240b4b537ae58d2522df4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/6274c56df14240b4b537ae58d2522df4 2023-04-06 17:57:51,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/028a9d58f2c74965b08d384a841e5409 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/028a9d58f2c74965b08d384a841e5409 2023-04-06 17:57:51,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fd8acaa520bd4b889d99ba68f8e4337b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/fd8acaa520bd4b889d99ba68f8e4337b 2023-04-06 17:57:51,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/08e524bf8b0f47f091cce5f56eadf6af to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/08e524bf8b0f47f091cce5f56eadf6af 2023-04-06 17:57:51,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/568b998d99db45f4b25297641df7eef7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/568b998d99db45f4b25297641df7eef7 2023-04-06 17:57:51,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/43ece17fb6524d6286588a433ae5877d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/43ece17fb6524d6286588a433ae5877d 2023-04-06 17:57:51,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/8f6d48066b4d4290bc11f26565aadaac to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/8f6d48066b4d4290bc11f26565aadaac 2023-04-06 17:57:51,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/0174cf20a46b4fa58d9665283ca46fcd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/0174cf20a46b4fa58d9665283ca46fcd 2023-04-06 17:57:51,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ae81a21bddca4391af57ba2139b58bd9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ae81a21bddca4391af57ba2139b58bd9 2023-04-06 17:57:51,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9f430e17211544b5a0fb72ee9f4620ab to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9f430e17211544b5a0fb72ee9f4620ab 2023-04-06 17:57:51,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/189738f5968e4ac491bba897cc1a9cf0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/189738f5968e4ac491bba897cc1a9cf0 2023-04-06 17:57:51,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/95834871f43143668b3d32f83108deb8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/95834871f43143668b3d32f83108deb8 2023-04-06 17:57:51,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/29131343b8fd456e87b2a45efd363cc6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/29131343b8fd456e87b2a45efd363cc6 2023-04-06 17:57:51,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/448d89f01bda48ccb28a86bba8de2422 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/448d89f01bda48ccb28a86bba8de2422 2023-04-06 17:57:51,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/09f7ff623c514553ab3794a29d7ca3ee to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/09f7ff623c514553ab3794a29d7ca3ee 2023-04-06 17:57:51,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f94951ca71444de99d5bd688f82ff1af to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/f94951ca71444de99d5bd688f82ff1af 2023-04-06 17:57:51,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d17e3748c1a841069429b0097a9cb120, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1e5a9718af0e4ae0a0fa698de8a43456, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/8202c5de29da4b8da3a1f4bb5a099568, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c4024bdc5f754fd4a3ca05d935f0a89b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/15983ac500214ac783854c3a50a685a5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c70722f61a72428288571d41dfd2c160, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/450dde3fc52f4555a9ecd747398c42a6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bed47ae9e4664d6db1c5e0c6cd4d73e7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bf624b96333f46438ee6b7a30c86ab7f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/4fdc4189a454480bbb0923c7f6b32046, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f1fedecc0a6e46b782ba60a5fff598ad, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f2f837f644934de2b924a0a7224bf164, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/5e2e3e9c577140379ca7ab713e3fd3da, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/970c9a2f3e2d4190a315a70e9a885d50, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/b0f4adaa34cb421faa3e820a7c403cb6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1658d8d69e62461bb95ebc3e6f4850a3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/54fb001a18b644f28e99d76ba66f2c41, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2e361ed2ced0402b8a1dd8e857778c7f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d38cd94c234f40c78d5768d1c86283da, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6b31eda630b5400792e330b089b73db1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3b7e4a7a1050418a825cafe4bbb4c54c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2713b47f392641bb90ebc3cf355861a2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3543610ddc3f471eb83c00109362d30f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/0d63a910e67d4221ae5893cd4a527275, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3222319d13fe462c96fa9b38b2f339cf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/a745c48669b8473ba9ce0043d20d1baf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/11e39c5275bd4200b8517c81bdd80c25, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bc3a474ce62840cc953ea4e304aebe9b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6541437a28c84dfc832e4cc73dbb6a2f] to archive 2023-04-06 17:57:51,388 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:57:51,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d17e3748c1a841069429b0097a9cb120 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d17e3748c1a841069429b0097a9cb120 2023-04-06 17:57:51,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1e5a9718af0e4ae0a0fa698de8a43456 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1e5a9718af0e4ae0a0fa698de8a43456 2023-04-06 17:57:51,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/8202c5de29da4b8da3a1f4bb5a099568 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/8202c5de29da4b8da3a1f4bb5a099568 2023-04-06 17:57:51,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c4024bdc5f754fd4a3ca05d935f0a89b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c4024bdc5f754fd4a3ca05d935f0a89b 2023-04-06 17:57:51,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/15983ac500214ac783854c3a50a685a5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/15983ac500214ac783854c3a50a685a5 2023-04-06 17:57:51,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c70722f61a72428288571d41dfd2c160 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/c70722f61a72428288571d41dfd2c160 2023-04-06 17:57:51,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/450dde3fc52f4555a9ecd747398c42a6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/450dde3fc52f4555a9ecd747398c42a6 2023-04-06 17:57:51,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bed47ae9e4664d6db1c5e0c6cd4d73e7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bed47ae9e4664d6db1c5e0c6cd4d73e7 2023-04-06 17:57:51,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bf624b96333f46438ee6b7a30c86ab7f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bf624b96333f46438ee6b7a30c86ab7f 2023-04-06 17:57:51,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/4fdc4189a454480bbb0923c7f6b32046 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/4fdc4189a454480bbb0923c7f6b32046 2023-04-06 17:57:51,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/7cd7f9cbc9c949ac8bed6bb9d6ac1c42 2023-04-06 17:57:51,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f1fedecc0a6e46b782ba60a5fff598ad to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f1fedecc0a6e46b782ba60a5fff598ad 2023-04-06 17:57:51,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f2f837f644934de2b924a0a7224bf164 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f2f837f644934de2b924a0a7224bf164 2023-04-06 17:57:51,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/5e2e3e9c577140379ca7ab713e3fd3da to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/5e2e3e9c577140379ca7ab713e3fd3da 2023-04-06 17:57:51,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/970c9a2f3e2d4190a315a70e9a885d50 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/970c9a2f3e2d4190a315a70e9a885d50 2023-04-06 17:57:51,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/b0f4adaa34cb421faa3e820a7c403cb6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/b0f4adaa34cb421faa3e820a7c403cb6 2023-04-06 17:57:51,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1658d8d69e62461bb95ebc3e6f4850a3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/1658d8d69e62461bb95ebc3e6f4850a3 2023-04-06 17:57:51,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/54fb001a18b644f28e99d76ba66f2c41 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/54fb001a18b644f28e99d76ba66f2c41 2023-04-06 17:57:51,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2e361ed2ced0402b8a1dd8e857778c7f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2e361ed2ced0402b8a1dd8e857778c7f 2023-04-06 17:57:51,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d38cd94c234f40c78d5768d1c86283da to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/d38cd94c234f40c78d5768d1c86283da 2023-04-06 17:57:51,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6b31eda630b5400792e330b089b73db1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6b31eda630b5400792e330b089b73db1 2023-04-06 17:57:51,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3b7e4a7a1050418a825cafe4bbb4c54c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3b7e4a7a1050418a825cafe4bbb4c54c 2023-04-06 17:57:51,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2713b47f392641bb90ebc3cf355861a2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/2713b47f392641bb90ebc3cf355861a2 2023-04-06 17:57:51,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3543610ddc3f471eb83c00109362d30f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3543610ddc3f471eb83c00109362d30f 2023-04-06 17:57:51,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/0d63a910e67d4221ae5893cd4a527275 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/0d63a910e67d4221ae5893cd4a527275 2023-04-06 17:57:51,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3222319d13fe462c96fa9b38b2f339cf to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/3222319d13fe462c96fa9b38b2f339cf 2023-04-06 17:57:51,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/a745c48669b8473ba9ce0043d20d1baf to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/a745c48669b8473ba9ce0043d20d1baf 2023-04-06 17:57:51,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/11e39c5275bd4200b8517c81bdd80c25 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/11e39c5275bd4200b8517c81bdd80c25 2023-04-06 17:57:51,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bc3a474ce62840cc953ea4e304aebe9b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/bc3a474ce62840cc953ea4e304aebe9b 2023-04-06 17:57:51,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6541437a28c84dfc832e4cc73dbb6a2f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/6541437a28c84dfc832e4cc73dbb6a2f 2023-04-06 17:57:51,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/aaaa6fee0e144465a7eef4e0fd4cdd30, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f99cca1745b42838c86258d1dd21cc3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6abf64169f76436a9e1621545b031c73, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/9e452c77694d4325be3f7d4932b4852a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/dc1db5d861a24a099c0e315026b6e92b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/644ede10c35744879754e8cb4b59f7de, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/01c281e9d04348fcaa131f45ca4e909c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/ae45cb3034004e97aa332874e47331e2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/2cc9ee31ac074d56887c6924591c4f90, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4323e8e6e6794ea5b2cdb10a7c362cfc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f1a716fdf8e4995be5eace3fbafdf37, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6e53ac5eacc84245bc026e5cbd99585d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/39de84c5833b4778b6f1b0ecfefaec23, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/429757045efe487f9085b5cef4b45952, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4274e15821ef4bec9cb76e06cff95377, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/50f9bc2762b441188fbc619992eb9011, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/048ce6d9eb654e54a51f16cb8d19318f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6853a518d69f403aaa35292b4ced1d22, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/15f88d53876a465e9118b1df277d3a9e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/bbe3e870ded847fa803030ecbf111037, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/43caa8fd6ab24951bacc8c6310fa56f0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/789783572af84f8aaf002b08bbab2695, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/3ec1e73a240a45f793b9ffca2a855ca4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/f8a32fc58d314d2bbe6b56c247102720, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b73353cd1a8543da8df5a255cd67ec46, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/26e5d799e1c64be8b3f62e22e3810f13, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/acc4cc7fe84e40a2a5061983c139e773, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/be038a9bed554aea8e4491f9a0c631b8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0b8fa16a0184452cadd474d297d9bca6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b78a6186cd6f4217b9c896d492c910a9] to archive 2023-04-06 17:57:51,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:57:51,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/aaaa6fee0e144465a7eef4e0fd4cdd30 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/aaaa6fee0e144465a7eef4e0fd4cdd30 2023-04-06 17:57:51,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f99cca1745b42838c86258d1dd21cc3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f99cca1745b42838c86258d1dd21cc3 2023-04-06 17:57:51,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6abf64169f76436a9e1621545b031c73 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6abf64169f76436a9e1621545b031c73 2023-04-06 17:57:51,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/9e452c77694d4325be3f7d4932b4852a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/9e452c77694d4325be3f7d4932b4852a 2023-04-06 17:57:51,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/dc1db5d861a24a099c0e315026b6e92b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/dc1db5d861a24a099c0e315026b6e92b 2023-04-06 17:57:51,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/644ede10c35744879754e8cb4b59f7de to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/644ede10c35744879754e8cb4b59f7de 2023-04-06 17:57:51,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/01c281e9d04348fcaa131f45ca4e909c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/01c281e9d04348fcaa131f45ca4e909c 2023-04-06 17:57:51,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/ae45cb3034004e97aa332874e47331e2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/ae45cb3034004e97aa332874e47331e2 2023-04-06 17:57:51,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/2cc9ee31ac074d56887c6924591c4f90 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/2cc9ee31ac074d56887c6924591c4f90 2023-04-06 17:57:51,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4323e8e6e6794ea5b2cdb10a7c362cfc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4323e8e6e6794ea5b2cdb10a7c362cfc 2023-04-06 17:57:51,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f1a716fdf8e4995be5eace3fbafdf37 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/1f1a716fdf8e4995be5eace3fbafdf37 2023-04-06 17:57:51,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6e53ac5eacc84245bc026e5cbd99585d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6e53ac5eacc84245bc026e5cbd99585d 2023-04-06 17:57:51,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/39de84c5833b4778b6f1b0ecfefaec23 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/39de84c5833b4778b6f1b0ecfefaec23 2023-04-06 17:57:51,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/429757045efe487f9085b5cef4b45952 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/429757045efe487f9085b5cef4b45952 2023-04-06 17:57:51,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4274e15821ef4bec9cb76e06cff95377 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/4274e15821ef4bec9cb76e06cff95377 2023-04-06 17:57:51,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/50f9bc2762b441188fbc619992eb9011 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/50f9bc2762b441188fbc619992eb9011 2023-04-06 17:57:51,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/048ce6d9eb654e54a51f16cb8d19318f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/048ce6d9eb654e54a51f16cb8d19318f 2023-04-06 17:57:51,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6853a518d69f403aaa35292b4ced1d22 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/6853a518d69f403aaa35292b4ced1d22 2023-04-06 17:57:51,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/15f88d53876a465e9118b1df277d3a9e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/15f88d53876a465e9118b1df277d3a9e 2023-04-06 17:57:51,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/bbe3e870ded847fa803030ecbf111037 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/bbe3e870ded847fa803030ecbf111037 2023-04-06 17:57:51,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/43caa8fd6ab24951bacc8c6310fa56f0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/43caa8fd6ab24951bacc8c6310fa56f0 2023-04-06 17:57:51,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/789783572af84f8aaf002b08bbab2695 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/789783572af84f8aaf002b08bbab2695 2023-04-06 17:57:51,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/3ec1e73a240a45f793b9ffca2a855ca4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/3ec1e73a240a45f793b9ffca2a855ca4 2023-04-06 17:57:51,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/f8a32fc58d314d2bbe6b56c247102720 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/f8a32fc58d314d2bbe6b56c247102720 2023-04-06 17:57:51,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b73353cd1a8543da8df5a255cd67ec46 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b73353cd1a8543da8df5a255cd67ec46 2023-04-06 17:57:51,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/26e5d799e1c64be8b3f62e22e3810f13 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/26e5d799e1c64be8b3f62e22e3810f13 2023-04-06 17:57:51,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/acc4cc7fe84e40a2a5061983c139e773 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/acc4cc7fe84e40a2a5061983c139e773 2023-04-06 17:57:51,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/be038a9bed554aea8e4491f9a0c631b8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/be038a9bed554aea8e4491f9a0c631b8 2023-04-06 17:57:51,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0b8fa16a0184452cadd474d297d9bca6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0b8fa16a0184452cadd474d297d9bca6 2023-04-06 17:57:51,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b78a6186cd6f4217b9c896d492c910a9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/b78a6186cd6f4217b9c896d492c910a9 2023-04-06 17:57:51,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/recovered.edits/552.seqid, newMaxSeqId=552, maxSeqId=1 2023-04-06 17:57:51,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05. 2023-04-06 17:57:51,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 9e540ddc475e6cca2398e6140b9c8e05: 2023-04-06 17:57:51,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:51,474 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=9e540ddc475e6cca2398e6140b9c8e05, regionState=CLOSED 2023-04-06 17:57:51,475 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803871474"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803871474"}]},"ts":"1680803871474"} 2023-04-06 17:57:51,476 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-06 17:57:51,477 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 9e540ddc475e6cca2398e6140b9c8e05, server=jenkins-hbase20.apache.org,38773,1680803776439 in 1.5290 sec 2023-04-06 17:57:51,478 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-06 17:57:51,478 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9e540ddc475e6cca2398e6140b9c8e05, UNASSIGN in 1.5340 sec 2023-04-06 17:57:51,478 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803871478"}]},"ts":"1680803871478"} 2023-04-06 17:57:51,479 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 17:57:51,480 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 17:57:51,482 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5440 sec 2023-04-06 17:57:52,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 17:57:52,045 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-06 17:57:52,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 17:57:52,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:52,048 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:52,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-06 17:57:52,049 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:52,052 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:52,054 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/recovered.edits] 2023-04-06 17:57:52,056 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/12c3c554813b40e88ca603d8d9ec5007 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/12c3c554813b40e88ca603d8d9ec5007 2023-04-06 17:57:52,057 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9001c723017049cc9c2957fa9b1d5a1a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/9001c723017049cc9c2957fa9b1d5a1a 2023-04-06 17:57:52,058 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ffeb5b0e257b4391bc995ddcb8fcd253 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/A/ffeb5b0e257b4391bc995ddcb8fcd253 2023-04-06 17:57:52,060 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/70283a2ab16a439094ebe75fcbdb32b8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/70283a2ab16a439094ebe75fcbdb32b8 2023-04-06 17:57:52,061 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/e93df4713a404ca4a1aebda8aa5fa4d2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/e93df4713a404ca4a1aebda8aa5fa4d2 2023-04-06 17:57:52,062 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f4950822a52c4ad8b50ddda05ff26cfb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/B/f4950822a52c4ad8b50ddda05ff26cfb 2023-04-06 17:57:52,063 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0db84a7d327f4c10835495d001446920 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/0db84a7d327f4c10835495d001446920 2023-04-06 17:57:52,064 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/710b45a53b024667a6f196ee4e9204d8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/710b45a53b024667a6f196ee4e9204d8 2023-04-06 17:57:52,065 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/895c6850c1ff4b40aa7319bdad489923 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/C/895c6850c1ff4b40aa7319bdad489923 2023-04-06 17:57:52,067 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/recovered.edits/552.seqid to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05/recovered.edits/552.seqid 2023-04-06 17:57:52,067 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/9e540ddc475e6cca2398e6140b9c8e05 2023-04-06 17:57:52,067 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:57:52,069 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:52,070 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 17:57:52,071 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 17:57:52,072 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:52,072 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 17:57:52,072 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680803872072"}]},"ts":"9223372036854775807"} 2023-04-06 17:57:52,074 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 17:57:52,074 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9e540ddc475e6cca2398e6140b9c8e05, NAME => 'TestAcidGuarantees,,1680803843199.9e540ddc475e6cca2398e6140b9c8e05.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 17:57:52,074 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 17:57:52,074 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680803872074"}]},"ts":"9223372036854775807"} 2023-04-06 17:57:52,075 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 17:57:52,076 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:52,077 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 30 msec 2023-04-06 17:57:52,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-06 17:57:52,150 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-06 17:57:52,163 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 227), OpenFileDescriptor=544 (was 559), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=450 (was 326) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=4406 (was 2967) - AvailableMemoryMB LEAK? - 2023-04-06 17:57:52,173 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=544, MaxFileDescriptor=60000, SystemLoadAverage=450, ProcessCount=166, AvailableMemoryMB=4405 2023-04-06 17:57:52,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:57:52,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:57:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:52,178 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 17:57:52,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-06 17:57:52,178 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 17:57:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 17:57:52,179 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:52,180 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f empty. 2023-04-06 17:57:52,180 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:52,180 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:57:52,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 17:57:52,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 17:57:52,596 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:57:52,599 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6db618d61b9573465a7386d437c4d03f, NAME => 'TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp 2023-04-06 17:57:52,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 17:57:53,004 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:57:53,005 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6db618d61b9573465a7386d437c4d03f, disabling compactions & flushes 2023-04-06 17:57:53,005 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,005 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,005 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. after waiting 0 ms 2023-04-06 17:57:53,005 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,005 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,005 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:57:53,007 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 17:57:53,008 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803873007"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803873007"}]},"ts":"1680803873007"} 2023-04-06 17:57:53,008 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 17:57:53,009 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 17:57:53,009 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803873009"}]},"ts":"1680803873009"} 2023-04-06 17:57:53,010 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 17:57:53,014 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, ASSIGN}] 2023-04-06 17:57:53,016 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, ASSIGN 2023-04-06 17:57:53,016 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=false 2023-04-06 17:57:53,168 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:53,168 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803873167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803873167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803873167"}]},"ts":"1680803873167"} 2023-04-06 17:57:53,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:57:53,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 17:57:53,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6db618d61b9573465a7386d437c4d03f, NAME => 'TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:57:53,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:57:53,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,325 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,326 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A 2023-04-06 17:57:53,326 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A 2023-04-06 17:57:53,326 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:53,326 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6db618d61b9573465a7386d437c4d03f columnFamilyName A 2023-04-06 17:57:53,327 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(310): Store=6db618d61b9573465a7386d437c4d03f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:53,327 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,328 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B 2023-04-06 17:57:53,328 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B 2023-04-06 17:57:53,328 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:53,328 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6db618d61b9573465a7386d437c4d03f columnFamilyName B 2023-04-06 17:57:53,329 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(310): Store=6db618d61b9573465a7386d437c4d03f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:53,329 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,330 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C 2023-04-06 17:57:53,330 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C 2023-04-06 17:57:53,330 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:53,330 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6db618d61b9573465a7386d437c4d03f columnFamilyName C 2023-04-06 17:57:53,331 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(310): Store=6db618d61b9573465a7386d437c4d03f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:53,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:57:53,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:53,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:57:53,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 6db618d61b9573465a7386d437c4d03f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66162001, jitterRate=-0.014109358191490173}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:57:53,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:57:53,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., pid=37, masterSystemTime=1680803873320 2023-04-06 17:57:53,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:53,339 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:53,339 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803873339"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803873339"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803873339"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803873339"}]},"ts":"1680803873339"} 2023-04-06 17:57:53,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-06 17:57:53,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 in 171 msec 2023-04-06 17:57:53,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-06 17:57:53,343 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, ASSIGN in 327 msec 2023-04-06 17:57:53,348 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 17:57:53,348 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803873348"}]},"ts":"1680803873348"} 2023-04-06 17:57:53,349 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 17:57:53,351 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 17:57:53,352 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1760 sec 2023-04-06 17:57:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 17:57:54,284 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-06 17:57:54,284 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x20312fe6 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:54,288 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fe8dd27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:54,289 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:54,291 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:54,292 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 17:57:54,293 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40836, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 17:57:54,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:57:54,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:57:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 17:57:54,707 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 17:57:54,708 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:57:54,708 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 17:57:54,710 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 17:57:54,713 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, REOPEN/MOVE}] 2023-04-06 17:57:54,714 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, REOPEN/MOVE 2023-04-06 17:57:54,715 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:54,715 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803874715"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803874715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803874715"}]},"ts":"1680803874715"} 2023-04-06 17:57:54,716 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:57:54,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:54,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 6db618d61b9573465a7386d437c4d03f, disabling compactions & flushes 2023-04-06 17:57:54,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:54,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:54,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. after waiting 0 ms 2023-04-06 17:57:54,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:54,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 17:57:54,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:54,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:57:54,871 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 6db618d61b9573465a7386d437c4d03f to self. 2023-04-06 17:57:54,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:54,873 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=CLOSED 2023-04-06 17:57:54,873 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803874873"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803874873"}]},"ts":"1680803874873"} 2023-04-06 17:57:54,875 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-06 17:57:54,875 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 in 158 msec 2023-04-06 17:57:54,875 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=true 2023-04-06 17:57:55,026 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,026 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803875026"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803875026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803875026"}]},"ts":"1680803875026"} 2023-04-06 17:57:55,027 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:57:55,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6db618d61b9573465a7386d437c4d03f, NAME => 'TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:57:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:57:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,183 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,184 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A 2023-04-06 17:57:55,184 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A 2023-04-06 17:57:55,184 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:55,185 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6db618d61b9573465a7386d437c4d03f columnFamilyName A 2023-04-06 17:57:55,186 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(310): Store=6db618d61b9573465a7386d437c4d03f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:55,186 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,187 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B 2023-04-06 17:57:55,187 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B 2023-04-06 17:57:55,187 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:55,187 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6db618d61b9573465a7386d437c4d03f columnFamilyName B 2023-04-06 17:57:55,188 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(310): Store=6db618d61b9573465a7386d437c4d03f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:55,188 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,188 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C 2023-04-06 17:57:55,188 DEBUG [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C 2023-04-06 17:57:55,188 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:57:55,189 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6db618d61b9573465a7386d437c4d03f columnFamilyName C 2023-04-06 17:57:55,189 INFO [StoreOpener-6db618d61b9573465a7386d437c4d03f-1] regionserver.HStore(310): Store=6db618d61b9573465a7386d437c4d03f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:57:55,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:55,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:57:55,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 6db618d61b9573465a7386d437c4d03f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69136928, jitterRate=0.030220508575439453}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:57:55,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:57:55,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., pid=42, masterSystemTime=1680803875179 2023-04-06 17:57:55,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:55,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:55,196 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,196 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803875196"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803875196"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803875196"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803875196"}]},"ts":"1680803875196"} 2023-04-06 17:57:55,198 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-06 17:57:55,198 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 in 170 msec 2023-04-06 17:57:55,199 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-06 17:57:55,199 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, REOPEN/MOVE in 485 msec 2023-04-06 17:57:55,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-06 17:57:55,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-06 17:57:55,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-06 17:57:55,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-06 17:57:55,203 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3a8219c3 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,207 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13655f6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,208 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x2ed5d233 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,211 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4339a2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,212 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x6a409627 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,215 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a849516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,216 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x6f29a773 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,222 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b6e51f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,223 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x48681e8f to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,227 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ad83ace, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,227 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3cb8db43 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,240 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@179d9850, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,241 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x1bd0918a to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,247 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61483370, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,248 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x5225f8f6 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,257 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4de67ff9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,258 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x144047c5 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,265 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695ffff3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,266 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x525c59d1 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:57:55,273 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44b44230, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:57:55,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:55,282 DEBUG [hconnection-0x454d2ad8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,282 DEBUG [hconnection-0xbe10855-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,282 DEBUG [hconnection-0x7c58cc78-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,282 DEBUG [hconnection-0xfdb4c4e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,283 DEBUG [hconnection-0x4e5224ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,287 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,287 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,288 DEBUG [hconnection-0x37ecc497-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,288 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,288 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,290 DEBUG [hconnection-0x391325f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,290 DEBUG [hconnection-0x337cf751-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:55,290 DEBUG [hconnection-0x7339806a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,292 DEBUG [hconnection-0x3c08e46c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:57:55,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:55,291 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,292 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:55,292 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:55,293 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:55,293 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:55,294 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:55,294 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42590, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,294 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:55,295 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,295 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,296 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42588, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,296 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42572, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:57:55,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:55,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:55,296 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:55,297 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,297 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:55,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:55,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:55,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:55,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:55,302 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:55,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:55,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:55,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:55,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:55,304 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:55,304 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:57:55,305 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:57:55,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:57:55,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:55,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:57:55,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:55,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:57:55,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:55,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:57:55,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:55,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803935318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803935318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803935319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803935320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803935320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803935421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803935421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803935421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803935423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803935423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803935624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803935624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803935624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803935625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803935625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,748 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406d99b9d4a331f4f9494e71b9a60a43bd6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d99b9d4a331f4f9494e71b9a60a43bd6 2023-04-06 17:57:55,767 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d8c1f6c5bc99482a89565d507bc72d9c 2023-04-06 17:57:55,780 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/25628d6a8fbd421b84c2fe306d2a1afe 2023-04-06 17:57:55,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803935928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803935929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803935931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803935932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:55,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:55,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803935934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,198 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/bc8eb6490cf64a5eba0b07f48cba3387 2023-04-06 17:57:56,202 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d8c1f6c5bc99482a89565d507bc72d9c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d8c1f6c5bc99482a89565d507bc72d9c 2023-04-06 17:57:56,207 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d8c1f6c5bc99482a89565d507bc72d9c, entries=150, sequenceid=16, filesize=25.1 K 2023-04-06 17:57:56,209 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/25628d6a8fbd421b84c2fe306d2a1afe as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/25628d6a8fbd421b84c2fe306d2a1afe 2023-04-06 17:57:56,220 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/25628d6a8fbd421b84c2fe306d2a1afe, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 17:57:56,221 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/bc8eb6490cf64a5eba0b07f48cba3387 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/bc8eb6490cf64a5eba0b07f48cba3387 2023-04-06 17:57:56,226 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/bc8eb6490cf64a5eba0b07f48cba3387, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 17:57:56,228 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6db618d61b9573465a7386d437c4d03f in 923ms, sequenceid=16, compaction requested=false 2023-04-06 17:57:56,228 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:57:56,228 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:56,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:56,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:56,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:56,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:56,230 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:56,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:56,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:56,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736d17a8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:56,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:56,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:56,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:56,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:56,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,233 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:56,233 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:56,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:56,237 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:56,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:56,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:56,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:56,237 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:56,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:56,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:56,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:56,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:56,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:56,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:56,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,243 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:56,243 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:56,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:56,344 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:56,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:56,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:56,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:56,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:56,350 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:56,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:56,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:56,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:56,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:56,351 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:56,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:56,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:56,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:56,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:56,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:56,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:56,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:56,354 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:56,354 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:57:56,354 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:57:56,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:57:56,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:56,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:57:56,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:56,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:57:56,355 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:56,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:56,434 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:57:56,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803936443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803936443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803936444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803936444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803936444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803936545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803936548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803936548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803936549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803936749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803936750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803936752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:56,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803936753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:56,769 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304061f3bd3bca00341a0b513b4735879886d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061f3bd3bca00341a0b513b4735879886d 2023-04-06 17:57:57,005 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 17:57:57,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803937054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803937055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803937057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803937059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,173 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/10c8b74d0e0142dea7c137dc77e7c294 2023-04-06 17:57:57,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803937454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803937561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803937564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803937565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:57,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803937567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,581 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/a2e3cc86cfe6437eaaf1f2da8645a211 2023-04-06 17:57:57,592 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/82282f9e44d74a54b1e666e1c5a6cade 2023-04-06 17:57:57,595 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/10c8b74d0e0142dea7c137dc77e7c294 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/10c8b74d0e0142dea7c137dc77e7c294 2023-04-06 17:57:57,599 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/10c8b74d0e0142dea7c137dc77e7c294, entries=150, sequenceid=36, filesize=25.1 K 2023-04-06 17:57:57,599 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/a2e3cc86cfe6437eaaf1f2da8645a211 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a2e3cc86cfe6437eaaf1f2da8645a211 2023-04-06 17:57:57,602 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a2e3cc86cfe6437eaaf1f2da8645a211, entries=150, sequenceid=36, filesize=11.6 K 2023-04-06 17:57:57,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/82282f9e44d74a54b1e666e1c5a6cade as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/82282f9e44d74a54b1e666e1c5a6cade 2023-04-06 17:57:57,606 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/82282f9e44d74a54b1e666e1c5a6cade, entries=150, sequenceid=36, filesize=11.6 K 2023-04-06 17:57:57,606 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6db618d61b9573465a7386d437c4d03f in 1252ms, sequenceid=36, compaction requested=false 2023-04-06 17:57:57,606 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:57:57,606 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:57,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:57,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:57,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:57,606 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:57,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:57,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f36407a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:57,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:57,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,610 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:57,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:57,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,610 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:57,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:57,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:57,611 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:57,611 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,611 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:57,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:57,613 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:57,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:57,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:57,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:57,613 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:57,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:57,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:57,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:57,620 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:57,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:57,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:57,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:57,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:57,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:57,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:57,620 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:57,620 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:57,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:57,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,621 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,621 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:57,621 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:57:57,720 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:57,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:57,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:57,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:57,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:57,723 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:57,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:57,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:57,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:57,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:57,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:57,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:57,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:57,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:57,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:57,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:57,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:57,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:57,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:57,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:57,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:57,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:57,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:57:57,726 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:57:57,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:57,738 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406a5682899683a4567b0aa14191aa91a21 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a5682899683a4567b0aa14191aa91a21 2023-04-06 17:57:58,144 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/5b443c757ba1422fb92a37604645ab7a 2023-04-06 17:57:58,561 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/4b60ebd1e867459987112cef15bc161b 2023-04-06 17:57:58,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:58,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:57:58,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803938598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803938601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803938602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803938606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803938707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803938707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803938708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803938709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803938915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803938916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803938917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803938919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,976 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/576795bdff16445aba827c1cb2c28316 2023-04-06 17:57:58,979 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/5b443c757ba1422fb92a37604645ab7a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5b443c757ba1422fb92a37604645ab7a 2023-04-06 17:57:58,981 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5b443c757ba1422fb92a37604645ab7a, entries=150, sequenceid=48, filesize=25.1 K 2023-04-06 17:57:58,982 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/4b60ebd1e867459987112cef15bc161b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/4b60ebd1e867459987112cef15bc161b 2023-04-06 17:57:58,985 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/4b60ebd1e867459987112cef15bc161b, entries=150, sequenceid=48, filesize=11.6 K 2023-04-06 17:57:58,986 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/576795bdff16445aba827c1cb2c28316 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/576795bdff16445aba827c1cb2c28316 2023-04-06 17:57:58,989 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/576795bdff16445aba827c1cb2c28316, entries=150, sequenceid=48, filesize=11.6 K 2023-04-06 17:57:58,990 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6db618d61b9573465a7386d437c4d03f in 1264ms, sequenceid=48, compaction requested=true 2023-04-06 17:57:58,990 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:57:58,990 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:57:58,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:58,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:58,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:57:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:57:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae26b98[Count = 0] remaining members to acquire global barrier 2023-04-06 17:57:58,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:57:58,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:58,993 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:58,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:58,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:57:58,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:57:58,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:57:58,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:57:58,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:57:58,995 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:58,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:57:58,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:57:58,998 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:57:58,998 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:57:58,998 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:57:58,998 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:57:58,998 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:57:58,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:58,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:58,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:57:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:57:58,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:59,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:59,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:57:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:57:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:59,001 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:57:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:57:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:59,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:57:59,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:57:59,022 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:57:59,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:59,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:59,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:57:59,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:59,022 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:57:59,022 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:57:59,023 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:59,023 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:57:59,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:59,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:57:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:57:59,122 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:57:59,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:57:59,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:57:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:57:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:57:59,125 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:57:59,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:59,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:57:59,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:57:59,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:59,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:59,126 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:57:59,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:57:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:57:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:59,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:57:59,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:57:59,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:57:59,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:57:59,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:57:59,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:57:59,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:57:59,129 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:57:59,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:57:59,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:57:59,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803939233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803939233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803939234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803939235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803939337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803939337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803939340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803939340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803939467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,472 DEBUG [Thread-2312] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:57:59,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803939542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,547 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040636df98ccc3404a0c93040b809a456c70 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040636df98ccc3404a0c93040b809a456c70 2023-04-06 17:57:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803939542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803939543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803939544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803939850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803939850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803939850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:57:59,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803939850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:57:59,952 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/b42f32c751d440979e9edc11857a2b17 2023-04-06 17:58:00,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:00,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803940353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:00,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803940356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:00,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803940357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,361 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/a8422fc40f524383adf0dd1997029b93 2023-04-06 17:58:00,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803940357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,372 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/052ca744eaa4417c91cc59e689ca9201 2023-04-06 17:58:00,375 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/b42f32c751d440979e9edc11857a2b17 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b42f32c751d440979e9edc11857a2b17 2023-04-06 17:58:00,379 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b42f32c751d440979e9edc11857a2b17, entries=150, sequenceid=68, filesize=25.1 K 2023-04-06 17:58:00,379 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/a8422fc40f524383adf0dd1997029b93 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a8422fc40f524383adf0dd1997029b93 2023-04-06 17:58:00,383 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a8422fc40f524383adf0dd1997029b93, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 17:58:00,383 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/052ca744eaa4417c91cc59e689ca9201 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/052ca744eaa4417c91cc59e689ca9201 2023-04-06 17:58:00,386 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/052ca744eaa4417c91cc59e689ca9201, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 17:58:00,387 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6db618d61b9573465a7386d437c4d03f in 1258ms, sequenceid=68, compaction requested=true 2023-04-06 17:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:00,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:00,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:00,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:00,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:00,388 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:00,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:00,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:00,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54c48bd1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:00,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:00,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:00,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:00,392 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:00,393 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:00,393 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:00,393 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:00,395 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:00,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:00,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:00,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:00,395 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:00,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:00,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:00,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:00,399 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,402 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:00,402 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:00,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:00,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:00,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:00,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:00,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:00,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:00,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:00,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,403 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:00,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:00,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,403 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:00,403 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:00,503 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:00,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:00,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:00,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:00,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:00,507 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:00,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:00,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:00,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:00,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,507 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:00,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,508 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:00,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:00,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:00,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:00,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:00,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:00,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:00,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:00,509 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:00,509 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:00,509 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:00,516 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:00,516 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:00,516 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:00,516 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:00,517 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:00,536 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ff0742276f4d465a9ca2d699f941d6bb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ff0742276f4d465a9ca2d699f941d6bb 2023-04-06 17:58:00,949 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=80, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d7f4ae8691914936aa8662318b96b95a 2023-04-06 17:58:00,966 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/acb5f02abc9d449baca069b6639812fc 2023-04-06 17:58:01,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:01,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:01,386 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/9323121315734ec0b25eada158b4cec6 2023-04-06 17:58:01,390 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d7f4ae8691914936aa8662318b96b95a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d7f4ae8691914936aa8662318b96b95a 2023-04-06 17:58:01,393 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d7f4ae8691914936aa8662318b96b95a, entries=150, sequenceid=80, filesize=25.1 K 2023-04-06 17:58:01,394 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/acb5f02abc9d449baca069b6639812fc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/acb5f02abc9d449baca069b6639812fc 2023-04-06 17:58:01,398 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/acb5f02abc9d449baca069b6639812fc, entries=150, sequenceid=80, filesize=11.6 K 2023-04-06 17:58:01,399 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/9323121315734ec0b25eada158b4cec6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/9323121315734ec0b25eada158b4cec6 2023-04-06 17:58:01,403 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/9323121315734ec0b25eada158b4cec6, entries=150, sequenceid=80, filesize=11.6 K 2023-04-06 17:58:01,403 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=93.93 KB/96180 for 6db618d61b9573465a7386d437c4d03f in 887ms, sequenceid=80, compaction requested=true 2023-04-06 17:58:01,403 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:01,403 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:01,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:01,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:01,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:01,404 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:01,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f0f16b6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:01,406 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:01,406 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,407 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:01,407 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:01,407 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:01,407 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,407 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:01,408 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:01,408 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,408 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:01,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:01,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:01,410 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:01,410 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:01,410 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:01,410 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:01,410 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:01,411 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,411 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,411 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,412 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:01,418 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:01,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:01,418 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:01,418 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:01,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:01,428 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:58:01,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:01,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:01,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:01,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:01,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:01,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:01,459 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304060a8b28bd807c4efa8f343c8565014e87 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060a8b28bd807c4efa8f343c8565014e87 2023-04-06 17:58:01,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803941450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803941451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803941453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803941458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,518 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:01,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:01,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:01,521 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:01,521 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,521 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:01,521 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:01,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,523 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:01,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:01,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:01,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:01,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:01,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:01,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:01,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:01,534 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,534 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:01,534 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:01,535 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:01,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d7aabd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:01,541 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:01,541 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:01,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,543 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:01,543 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,543 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:01,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:01,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:01,543 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:01,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:01,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:01,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:01,543 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:01,544 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,544 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,544 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:01,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,548 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:01,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,548 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:01,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803941564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803941565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803941565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803941572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,649 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:01,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:01,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:01,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:01,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:01,654 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:01,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:01,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:01,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,655 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:01,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:01,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:01,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:01,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:01,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:01,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:01,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:01,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:01,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,672 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b20bbd4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:01,675 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:01,675 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,676 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:01,676 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,676 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:01,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:01,676 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:01,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:01,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,677 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,677 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:01,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:01,681 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:01,681 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:01,681 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:01,681 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:01,681 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:01,682 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,682 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,683 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,683 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,691 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,698 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,698 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:01,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:01,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:01,698 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,699 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:01,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,699 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,699 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:01,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803941771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803941772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803941772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:01,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803941779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,800 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:01,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:01,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:01,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:01,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:01,806 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:01,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:01,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:01,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:01,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:01,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:01,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:01,814 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:01,814 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,814 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:01,815 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:01,815 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:01,816 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,816 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:01,816 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:01,816 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:01,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:01,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:01,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:01,824 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,824 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,825 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,825 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:01,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4a2f62[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:01,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:01,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,830 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:01,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,832 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:01,831 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:01,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:01,836 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:01,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:01,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:01,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:01,836 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:01,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,840 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,840 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:01,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:01,840 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:01,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:01,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:01,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:01,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:01,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:01,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:01,846 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:01,846 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:01,873 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d4344136d307420688a8520e2ba6854d 2023-04-06 17:58:01,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/ae768ba8cfe7473db7d8779f5d9814e0 2023-04-06 17:58:01,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/55cdfc8656e84e7bacd82961a7a35dbd 2023-04-06 17:58:01,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d4344136d307420688a8520e2ba6854d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d4344136d307420688a8520e2ba6854d 2023-04-06 17:58:01,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d4344136d307420688a8520e2ba6854d, entries=250, sequenceid=99, filesize=38.7 K 2023-04-06 17:58:01,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/ae768ba8cfe7473db7d8779f5d9814e0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ae768ba8cfe7473db7d8779f5d9814e0 2023-04-06 17:58:01,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ae768ba8cfe7473db7d8779f5d9814e0, entries=150, sequenceid=99, filesize=11.6 K 2023-04-06 17:58:01,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/55cdfc8656e84e7bacd82961a7a35dbd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/55cdfc8656e84e7bacd82961a7a35dbd 2023-04-06 17:58:01,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/55cdfc8656e84e7bacd82961a7a35dbd, entries=150, sequenceid=99, filesize=11.6 K 2023-04-06 17:58:01,945 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6db618d61b9573465a7386d437c4d03f in 517ms, sequenceid=99, compaction requested=true 2023-04-06 17:58:01,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:01,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:01,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:58:01,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:01,945 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:58:01,946 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:01,946 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:01,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:01,947 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:58:01,947 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 168360 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:58:01,947 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/B is initiating minor compaction (all files) 2023-04-06 17:58:01,947 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/A is initiating minor compaction (all files) 2023-04-06 17:58:01,947 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/B in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,947 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/A in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,947 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d8c1f6c5bc99482a89565d507bc72d9c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/10c8b74d0e0142dea7c137dc77e7c294, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5b443c757ba1422fb92a37604645ab7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b42f32c751d440979e9edc11857a2b17, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d7f4ae8691914936aa8662318b96b95a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d4344136d307420688a8520e2ba6854d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=164.4 K 2023-04-06 17:58:01,947 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/25628d6a8fbd421b84c2fe306d2a1afe, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a2e3cc86cfe6437eaaf1f2da8645a211, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/4b60ebd1e867459987112cef15bc161b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a8422fc40f524383adf0dd1997029b93, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/acb5f02abc9d449baca069b6639812fc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ae768ba8cfe7473db7d8779f5d9814e0] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=69.6 K 2023-04-06 17:58:01,948 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d8c1f6c5bc99482a89565d507bc72d9c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680803875299 2023-04-06 17:58:01,948 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 25628d6a8fbd421b84c2fe306d2a1afe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680803875299 2023-04-06 17:58:01,948 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 10c8b74d0e0142dea7c137dc77e7c294, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680803875317 2023-04-06 17:58:01,948 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a2e3cc86cfe6437eaaf1f2da8645a211, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680803875317 2023-04-06 17:58:01,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:01,948 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 5b443c757ba1422fb92a37604645ab7a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803876440 2023-04-06 17:58:01,948 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4b60ebd1e867459987112cef15bc161b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803876440 2023-04-06 17:58:01,949 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b42f32c751d440979e9edc11857a2b17, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680803878600 2023-04-06 17:58:01,949 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a8422fc40f524383adf0dd1997029b93, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680803878600 2023-04-06 17:58:01,949 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d7f4ae8691914936aa8662318b96b95a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680803879226 2023-04-06 17:58:01,949 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting acb5f02abc9d449baca069b6639812fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680803879226 2023-04-06 17:58:01,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:01,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:01,949 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:01,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:01,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:01,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,950 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d4344136d307420688a8520e2ba6854d, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680803881394 2023-04-06 17:58:01,950 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting ae768ba8cfe7473db7d8779f5d9814e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680803881394 2023-04-06 17:58:01,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:01,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:01,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:01,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:01,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:01,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:01,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:01,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:01,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:01,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:01,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:01,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:01,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:01,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:01,954 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:58:01,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:01,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:01,954 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:01,955 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:01,955 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:01,955 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:01,968 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#B#compaction#323 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:01,989 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#A#compaction#322 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:02,005 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/c6d9b1d9ce1e44d5b94eb8ff6b002049 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c6d9b1d9ce1e44d5b94eb8ff6b002049 2023-04-06 17:58:02,010 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/B of 6db618d61b9573465a7386d437c4d03f into c6d9b1d9ce1e44d5b94eb8ff6b002049(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:02,010 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:02,010 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/B, priority=10, startTime=1680803881945; duration=0sec 2023-04-06 17:58:02,010 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:02,010 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:58:02,011 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:58:02,011 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/C is initiating minor compaction (all files) 2023-04-06 17:58:02,012 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/C in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,012 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/bc8eb6490cf64a5eba0b07f48cba3387, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/82282f9e44d74a54b1e666e1c5a6cade, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/576795bdff16445aba827c1cb2c28316, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/052ca744eaa4417c91cc59e689ca9201, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/9323121315734ec0b25eada158b4cec6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/55cdfc8656e84e7bacd82961a7a35dbd] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=69.6 K 2023-04-06 17:58:02,012 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting bc8eb6490cf64a5eba0b07f48cba3387, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680803875299 2023-04-06 17:58:02,012 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 82282f9e44d74a54b1e666e1c5a6cade, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680803875317 2023-04-06 17:58:02,012 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 576795bdff16445aba827c1cb2c28316, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803876440 2023-04-06 17:58:02,013 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 052ca744eaa4417c91cc59e689ca9201, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680803878600 2023-04-06 17:58:02,013 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9323121315734ec0b25eada158b4cec6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680803879226 2023-04-06 17:58:02,013 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 55cdfc8656e84e7bacd82961a7a35dbd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680803881394 2023-04-06 17:58:02,048 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#C#compaction#325 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:02,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:02,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:02,115 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/83858d2ef09b494aa9a7d561b097e50b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/83858d2ef09b494aa9a7d561b097e50b 2023-04-06 17:58:02,119 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/C of 6db618d61b9573465a7386d437c4d03f into 83858d2ef09b494aa9a7d561b097e50b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:02,119 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:02,119 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/C, priority=10, startTime=1680803881946; duration=0sec 2023-04-06 17:58:02,120 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:02,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803942124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803942124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803942125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803942140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803942241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803942241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803942241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803942251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,388 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062f88e50ff6e545ba83c3e096c727664a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f88e50ff6e545ba83c3e096c727664a 2023-04-06 17:58:02,395 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/1c483b6da7784fb29c7e6eca59372202 2023-04-06 17:58:02,408 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/93d1d0386f90451ba315db386d58a10f 2023-04-06 17:58:02,424 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/f100261c31dd4da2bc9c7e5ac33a281d 2023-04-06 17:58:02,428 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/1c483b6da7784fb29c7e6eca59372202 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/1c483b6da7784fb29c7e6eca59372202 2023-04-06 17:58:02,432 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/1c483b6da7784fb29c7e6eca59372202, entries=150, sequenceid=112, filesize=25.1 K 2023-04-06 17:58:02,433 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/93d1d0386f90451ba315db386d58a10f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/93d1d0386f90451ba315db386d58a10f 2023-04-06 17:58:02,440 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/93d1d0386f90451ba315db386d58a10f, entries=150, sequenceid=112, filesize=11.6 K 2023-04-06 17:58:02,440 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/f100261c31dd4da2bc9c7e5ac33a281d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/f100261c31dd4da2bc9c7e5ac33a281d 2023-04-06 17:58:02,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803942450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803942450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,453 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/f100261c31dd4da2bc9c7e5ac33a281d, entries=150, sequenceid=112, filesize=11.6 K 2023-04-06 17:58:02,453 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6db618d61b9573465a7386d437c4d03f in 499ms, sequenceid=112, compaction requested=false 2023-04-06 17:58:02,453 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:02,454 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:02,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:02,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:02,454 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:02,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,455 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:02,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c47201b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:02,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:02,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,457 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:02,457 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:02,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:02,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:02,457 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:58:02,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:02,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:02,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:02,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:02,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:02,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:02,460 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:02,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:02,463 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:02,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:02,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:02,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:02,463 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:02,472 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,472 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,472 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,473 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,474 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:02,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:02,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:02,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,481 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,482 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,482 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,482 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:02,482 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:02,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803942497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803942497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,518 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/abc619725b6344cba66e2924c1548340 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/abc619725b6344cba66e2924c1548340 2023-04-06 17:58:02,523 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/A of 6db618d61b9573465a7386d437c4d03f into abc619725b6344cba66e2924c1548340(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:02,523 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:02,523 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/A, priority=10, startTime=1680803881945; duration=0sec 2023-04-06 17:58:02,523 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:02,582 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:02,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:02,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:02,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:02,593 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:02,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:02,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:02,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,595 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:02,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:02,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:02,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:02,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:02,598 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,598 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:02,598 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:02,598 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:02,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,599 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:02,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335ae56c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:02,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:02,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,601 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:02,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:02,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:02,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:02,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,602 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,602 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:02,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:02,603 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:02,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:02,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:02,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:02,604 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:02,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:02,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:02,608 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:02,608 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:02,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:02,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,609 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,609 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,609 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,609 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803942604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803942604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,708 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:02,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:02,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:02,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:02,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:02,710 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:02,711 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,711 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:02,711 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:02,711 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,711 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,712 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:02,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:02,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:02,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:02,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:02,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:02,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:02,713 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,713 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:02,714 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:02,714 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:02,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:02,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13732452[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:02,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:02,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:02,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,720 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:02,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:02,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:02,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:02,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:02,725 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:02,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:02,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:02,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:02,725 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:02,727 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:02,727 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,727 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:02,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,736 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:02,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,740 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:02,740 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:02,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803942753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803942753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803942818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:02,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803942818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,841 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:02,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:02,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:02,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:02,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:02,849 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:02,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:02,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:02,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,851 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,851 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:02,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:02,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:02,855 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:02,855 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,855 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:02,855 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:02,855 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:02,855 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,855 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:02,855 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:02,855 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,855 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:02,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:02,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:02,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:02,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:02,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dca16c8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:02,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:02,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,866 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:02,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:02,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:02,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:02,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:02,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:02,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:02,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:02,870 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:02,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:02,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:02,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:02,870 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:02,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:02,871 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,872 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:02,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:02,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:02,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:02,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:02,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:02,875 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:02,875 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,875 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:02,875 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:02,890 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040681b05793cc8b4de3b3410e16bd4b19c5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040681b05793cc8b4de3b3410e16bd4b19c5 2023-04-06 17:58:02,975 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:02,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:02,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:02,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:02,978 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:02,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:02,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:02,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:02,979 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:02,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:02,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:02,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:02,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:02,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:02,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:02,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:02,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:02,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:02,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:02,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:02,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:02,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:02,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:02,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:02,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:02,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:02,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:03,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1731032c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:03,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:03,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:03,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:03,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:03,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,012 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:03,012 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:03,013 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:03,013 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:03,013 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:03,013 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:03,013 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:03,014 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,014 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,014 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,015 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,018 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:03,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:03,021 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:03,021 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:03,020 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:03,121 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:03,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:03,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:03,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803943121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803943122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:03,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:03,126 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:03,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:03,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:03,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,127 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:03,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:03,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:03,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:03,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:03,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:03,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:03,138 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:03,138 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:03,138 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:03,138 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:03,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:03,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:03,138 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:03,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:03,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263be29[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:03,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:03,141 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,142 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:03,142 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,142 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:03,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:03,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:03,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:03,146 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:03,146 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:03,146 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:03,146 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:03,146 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:03,147 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,147 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,148 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,148 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:03,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:03,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,151 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,151 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,151 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:03,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,151 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:03,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:03,251 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:03,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:03,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:03,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:03,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:03,258 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:03,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:03,258 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:03,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,260 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,261 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,261 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,261 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:03,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:03,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:03,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:03,263 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:03,263 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,263 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:03,265 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:03,265 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:03,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:03,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:03,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:03,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:03,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:03,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:03,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803943264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803943268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:03,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66764a8c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:03,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:03,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,282 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:03,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:03,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:03,283 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,283 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:03,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:03,288 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:03,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:03,291 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:03,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:03,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:03,291 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:03,291 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:03,298 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,298 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,299 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,301 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,304 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,304 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:03,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:03,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,305 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:03,305 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:03,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,306 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/b4331337ac0b472ead6849d82b8d57d5 2023-04-06 17:58:03,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:03,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/a5dfbe3de5f0489788aca9d6312d15cc 2023-04-06 17:58:03,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/a2dc8800ebe74c17811feaf5a254b7d5 2023-04-06 17:58:03,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/b4331337ac0b472ead6849d82b8d57d5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b4331337ac0b472ead6849d82b8d57d5 2023-04-06 17:58:03,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b4331337ac0b472ead6849d82b8d57d5, entries=200, sequenceid=134, filesize=32.1 K 2023-04-06 17:58:03,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/a5dfbe3de5f0489788aca9d6312d15cc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a5dfbe3de5f0489788aca9d6312d15cc 2023-04-06 17:58:03,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a5dfbe3de5f0489788aca9d6312d15cc, entries=150, sequenceid=134, filesize=11.7 K 2023-04-06 17:58:03,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/a2dc8800ebe74c17811feaf5a254b7d5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/a2dc8800ebe74c17811feaf5a254b7d5 2023-04-06 17:58:03,405 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:03,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:03,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:03,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/a2dc8800ebe74c17811feaf5a254b7d5, entries=150, sequenceid=134, filesize=11.7 K 2023-04-06 17:58:03,428 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6db618d61b9573465a7386d437c4d03f in 970ms, sequenceid=134, compaction requested=true 2023-04-06 17:58:03,428 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:03,428 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:03,428 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:03,428 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:03,428 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:03,428 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:03,430 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84534 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:03,430 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:03,430 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/A is initiating minor compaction (all files) 2023-04-06 17:58:03,430 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/B is initiating minor compaction (all files) 2023-04-06 17:58:03,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:03,430 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/B in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,430 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/A in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:03,430 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/abc619725b6344cba66e2924c1548340, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/1c483b6da7784fb29c7e6eca59372202, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b4331337ac0b472ead6849d82b8d57d5] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=82.6 K 2023-04-06 17:58:03,430 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:03,430 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c6d9b1d9ce1e44d5b94eb8ff6b002049, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/93d1d0386f90451ba315db386d58a10f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a5dfbe3de5f0489788aca9d6312d15cc] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=35.1 K 2023-04-06 17:58:03,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:03,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:03,430 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting abc619725b6344cba66e2924c1548340, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680803881394 2023-04-06 17:58:03,431 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c6d9b1d9ce1e44d5b94eb8ff6b002049, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680803881394 2023-04-06 17:58:03,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,431 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 1c483b6da7784fb29c7e6eca59372202, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680803881441 2023-04-06 17:58:03,431 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 93d1d0386f90451ba315db386d58a10f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680803881441 2023-04-06 17:58:03,431 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b4331337ac0b472ead6849d82b8d57d5, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803882105 2023-04-06 17:58:03,431 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a5dfbe3de5f0489788aca9d6312d15cc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803882105 2023-04-06 17:58:03,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,432 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:03,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:03,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:03,447 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#B#compaction#331 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:03,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:03,466 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:03,466 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,466 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:03,466 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:58:03,466 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:03,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:03,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:03,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:03,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:03,467 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:03,488 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#A#compaction#332 average throughput is 0.58 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:03,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:03,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:03,568 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406395fa00e3ef947549923d11f9ebc8e24 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406395fa00e3ef947549923d11f9ebc8e24 2023-04-06 17:58:03,594 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/b5d3ae88d9ec40d0ba99f7cff18bb291 2023-04-06 17:58:03,610 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/82498514a0e740ed84049609480666d4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/82498514a0e740ed84049609480666d4 2023-04-06 17:58:03,616 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/A of 6db618d61b9573465a7386d437c4d03f into 82498514a0e740ed84049609480666d4(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:03,616 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:03,616 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/A, priority=13, startTime=1680803883428; duration=0sec 2023-04-06 17:58:03,616 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:03,616 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:03,617 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:03,617 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/C is initiating minor compaction (all files) 2023-04-06 17:58:03,617 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/C in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,618 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/83858d2ef09b494aa9a7d561b097e50b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/f100261c31dd4da2bc9c7e5ac33a281d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/a2dc8800ebe74c17811feaf5a254b7d5] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=35.1 K 2023-04-06 17:58:03,618 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 83858d2ef09b494aa9a7d561b097e50b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680803881394 2023-04-06 17:58:03,618 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f100261c31dd4da2bc9c7e5ac33a281d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680803881441 2023-04-06 17:58:03,618 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting a2dc8800ebe74c17811feaf5a254b7d5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803882105 2023-04-06 17:58:03,628 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#C#compaction#335 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:03,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803943636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803943640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803943641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,650 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/f2e22fc438794a948bcf7b54be1d7d2b 2023-04-06 17:58:03,690 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/99f1048629ac47c7be9091624105831e 2023-04-06 17:58:03,695 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/b5d3ae88d9ec40d0ba99f7cff18bb291 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b5d3ae88d9ec40d0ba99f7cff18bb291 2023-04-06 17:58:03,706 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b5d3ae88d9ec40d0ba99f7cff18bb291, entries=150, sequenceid=147, filesize=25.3 K 2023-04-06 17:58:03,707 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/f2e22fc438794a948bcf7b54be1d7d2b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f2e22fc438794a948bcf7b54be1d7d2b 2023-04-06 17:58:03,718 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f2e22fc438794a948bcf7b54be1d7d2b, entries=150, sequenceid=147, filesize=11.8 K 2023-04-06 17:58:03,719 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/99f1048629ac47c7be9091624105831e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/99f1048629ac47c7be9091624105831e 2023-04-06 17:58:03,723 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/99f1048629ac47c7be9091624105831e, entries=150, sequenceid=147, filesize=11.8 K 2023-04-06 17:58:03,724 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6db618d61b9573465a7386d437c4d03f in 257ms, sequenceid=147, compaction requested=false 2023-04-06 17:58:03,724 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:03,724 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:03,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:03,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:03,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:03,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,725 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,727 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,727 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:03,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b876cdc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:03,727 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:03,727 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,728 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:03,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:03,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:03,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:03,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:03,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:03,738 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:03,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:03,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:03,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:03,738 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:03,747 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,747 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:03,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-06 17:58:03,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:03,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:03,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:03,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:03,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:03,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,751 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:03,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:03,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:03,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,757 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:03,757 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:03,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803943773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803943780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803943783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,788 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406013d04fcfdac48ebbef9a3777a25986b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406013d04fcfdac48ebbef9a3777a25986b 2023-04-06 17:58:03,811 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/c1dd342ba6a34d748e2c83970d7573d8 2023-04-06 17:58:03,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/b44e6fa5483c40b5b51905e75d199a7a 2023-04-06 17:58:03,860 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:03,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:03,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:03,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:03,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:03,866 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:03,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:03,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:03,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,867 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:03,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:03,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:03,901 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:03,901 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:03,901 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:03,902 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:03,902 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:03,905 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/c36dc6d806d74bd19f254601335e4017 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c36dc6d806d74bd19f254601335e4017 2023-04-06 17:58:03,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803943900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803943901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,902 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:03,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:03,907 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:03,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803943901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:03,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/c3c212775e98489b939312213be3d287 2023-04-06 17:58:03,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,909 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:03,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74303ae3[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:03,911 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:03,911 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,912 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/B of 6db618d61b9573465a7386d437c4d03f into c36dc6d806d74bd19f254601335e4017(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:03,912 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:03,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:03,912 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,912 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/B, priority=13, startTime=1680803883428; duration=0sec 2023-04-06 17:58:03,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:03,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:03,912 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:03,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:03,913 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:03,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:03,915 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:03,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:03,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:03,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:03,915 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:03,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:03,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:03,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:03,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:03,920 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/c1dd342ba6a34d748e2c83970d7573d8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c1dd342ba6a34d748e2c83970d7573d8 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:03,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:03,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:03,923 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:03,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:03,923 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:03,923 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:03,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:03,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:03,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c1dd342ba6a34d748e2c83970d7573d8, entries=200, sequenceid=169, filesize=32.1 K 2023-04-06 17:58:03,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/b44e6fa5483c40b5b51905e75d199a7a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/b44e6fa5483c40b5b51905e75d199a7a 2023-04-06 17:58:03,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/b44e6fa5483c40b5b51905e75d199a7a, entries=150, sequenceid=169, filesize=11.8 K 2023-04-06 17:58:03,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/c3c212775e98489b939312213be3d287 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c3c212775e98489b939312213be3d287 2023-04-06 17:58:03,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c3c212775e98489b939312213be3d287, entries=150, sequenceid=169, filesize=11.8 K 2023-04-06 17:58:03,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 6db618d61b9573465a7386d437c4d03f in 191ms, sequenceid=169, compaction requested=true 2023-04-06 17:58:03,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:03,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:03,940 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:03,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:03,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:58:03,950 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84936 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:03,950 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/A is initiating minor compaction (all files) 2023-04-06 17:58:03,951 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/A in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:03,951 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/82498514a0e740ed84049609480666d4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b5d3ae88d9ec40d0ba99f7cff18bb291, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c1dd342ba6a34d748e2c83970d7573d8] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=82.9 K 2023-04-06 17:58:03,951 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 82498514a0e740ed84049609480666d4, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803882105 2023-04-06 17:58:03,951 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b5d3ae88d9ec40d0ba99f7cff18bb291, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680803882487 2023-04-06 17:58:03,952 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c1dd342ba6a34d748e2c83970d7573d8, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803883605 2023-04-06 17:58:03,964 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#A#compaction#340 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:04,024 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:04,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:04,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:04,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:04,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:04,035 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:04,035 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:04,035 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:04,035 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:04,035 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:04,035 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:04,036 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:04,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:04,036 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:04,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:04,037 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,037 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:04,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:04,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:04,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:04,048 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:04,048 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:04,048 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:04,050 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:04,050 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:04,050 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:04,050 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:04,051 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:58:04,052 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:04,052 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:04,052 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:04,052 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:04,052 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:04,053 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:04,081 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/e05184ac796f4c85b89706ba793b4312 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e05184ac796f4c85b89706ba793b4312 2023-04-06 17:58:04,086 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/C of 6db618d61b9573465a7386d437c4d03f into e05184ac796f4c85b89706ba793b4312(size=12.0 K), total size for store is 35.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:04,086 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:04,086 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/C, priority=13, startTime=1680803883428; duration=0sec 2023-04-06 17:58:04,086 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:58:04,086 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:04,087 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:04,087 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/B is initiating minor compaction (all files) 2023-04-06 17:58:04,087 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/B in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:04,087 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c36dc6d806d74bd19f254601335e4017, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f2e22fc438794a948bcf7b54be1d7d2b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/b44e6fa5483c40b5b51905e75d199a7a] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=35.5 K 2023-04-06 17:58:04,088 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c36dc6d806d74bd19f254601335e4017, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803882105 2023-04-06 17:58:04,088 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f2e22fc438794a948bcf7b54be1d7d2b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680803882487 2023-04-06 17:58:04,088 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b44e6fa5483c40b5b51905e75d199a7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803883634 2023-04-06 17:58:04,097 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#B#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:04,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:04,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:04,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680803884175, max timestamp=1680803884207Num uniques 859; 2023-04-06 17:58:04,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 17:58:04,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803944213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803944220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803944221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803944270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803944278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803944322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803944330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803944331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,422 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/faa479294b4341dc9839049c6c4093a1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/faa479294b4341dc9839049c6c4093a1 2023-04-06 17:58:04,426 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/A of 6db618d61b9573465a7386d437c4d03f into faa479294b4341dc9839049c6c4093a1(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:04,426 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:04,427 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/A, priority=13, startTime=1680803883940; duration=0sec 2023-04-06 17:58:04,427 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:04,427 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:04,428 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:04,428 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/C is initiating minor compaction (all files) 2023-04-06 17:58:04,428 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/C in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:04,428 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e05184ac796f4c85b89706ba793b4312, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/99f1048629ac47c7be9091624105831e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c3c212775e98489b939312213be3d287] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=35.5 K 2023-04-06 17:58:04,428 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e05184ac796f4c85b89706ba793b4312, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680803882105 2023-04-06 17:58:04,428 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 99f1048629ac47c7be9091624105831e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680803882487 2023-04-06 17:58:04,429 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c3c212775e98489b939312213be3d287, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803883634 2023-04-06 17:58:04,451 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#C#compaction#343 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:04,482 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/750b487a3a624c6099d3e29c9a92c7cf as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/750b487a3a624c6099d3e29c9a92c7cf 2023-04-06 17:58:04,486 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/C of 6db618d61b9573465a7386d437c4d03f into 750b487a3a624c6099d3e29c9a92c7cf(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:04,486 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:04,486 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/C, priority=13, startTime=1680803883941; duration=0sec 2023-04-06 17:58:04,486 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:04,526 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c6b95749dbbb4632a0c903f1add0505e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c6b95749dbbb4632a0c903f1add0505e 2023-04-06 17:58:04,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803944528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803944539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803944540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,552 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/cc3f044c890742bfb4661ef3407d01e9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/cc3f044c890742bfb4661ef3407d01e9 2023-04-06 17:58:04,558 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/B of 6db618d61b9573465a7386d437c4d03f into cc3f044c890742bfb4661ef3407d01e9(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:04,558 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:04,558 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/B, priority=13, startTime=1680803883940; duration=0sec 2023-04-06 17:58:04,558 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:04,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803944833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803944845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:04,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803944847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:04,960 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/f9dbda42277746589395829912785b86 2023-04-06 17:58:04,991 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/3c071cec64544f81b3ac8c32c54e5674 2023-04-06 17:58:05,014 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/cd9fb605048445369a3431742d48b39e 2023-04-06 17:58:05,022 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/f9dbda42277746589395829912785b86 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/f9dbda42277746589395829912785b86 2023-04-06 17:58:05,032 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/f9dbda42277746589395829912785b86, entries=100, sequenceid=181, filesize=18.5 K 2023-04-06 17:58:05,033 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/3c071cec64544f81b3ac8c32c54e5674 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/3c071cec64544f81b3ac8c32c54e5674 2023-04-06 17:58:05,045 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/3c071cec64544f81b3ac8c32c54e5674, entries=100, sequenceid=181, filesize=9.4 K 2023-04-06 17:58:05,046 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/cd9fb605048445369a3431742d48b39e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/cd9fb605048445369a3431742d48b39e 2023-04-06 17:58:05,052 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/cd9fb605048445369a3431742d48b39e, entries=100, sequenceid=181, filesize=9.4 K 2023-04-06 17:58:05,052 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=120.76 KB/123660 for 6db618d61b9573465a7386d437c4d03f in 1001ms, sequenceid=181, compaction requested=false 2023-04-06 17:58:05,052 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:05,052 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:05,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:05,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:05,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:05,053 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:05,054 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,055 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,055 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:05,056 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:05,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:05,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bbd50ba[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:05,058 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:05,058 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,059 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,059 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:05,059 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:05,059 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:05,059 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,059 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:05,060 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:05,060 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:05,060 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:05,061 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:05,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:05,067 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:05,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:05,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:05,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:05,067 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:05,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:05,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:05,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:05,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:05,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:05,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:05,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:05,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:05,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:05,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:05,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:05,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:05,084 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:05,084 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:05,184 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:05,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:05,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:05,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:05,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:05,188 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:05,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:05,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:05,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:05,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:05,189 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:05,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:05,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:05,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:05,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:05,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:05,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:05,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:05,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:05,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:05,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:05,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:05,190 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:05,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:05,191 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=120.76 KB heapSize=351.95 KB 2023-04-06 17:58:05,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:05,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:05,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:05,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:05,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:05,191 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:05,221 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b27bffa6d2684ca6a99b0feece8bebc3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b27bffa6d2684ca6a99b0feece8bebc3 2023-04-06 17:58:05,242 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=210, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/0de21d62611b443ea972eee8ce34ad9b 2023-04-06 17:58:05,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:05,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:05,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803945357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803945358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803945359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803945465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803945465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803945470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,672 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/60027ba21a48417c9ade85539f4f9657 2023-04-06 17:58:05,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803945669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803945670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803945678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803945976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803945977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:05,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:05,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803945986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,080 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/48e5c7645d48465b977c39b9440fb704 2023-04-06 17:58:06,085 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/0de21d62611b443ea972eee8ce34ad9b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/0de21d62611b443ea972eee8ce34ad9b 2023-04-06 17:58:06,089 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/0de21d62611b443ea972eee8ce34ad9b, entries=150, sequenceid=210, filesize=25.3 K 2023-04-06 17:58:06,090 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/60027ba21a48417c9ade85539f4f9657 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/60027ba21a48417c9ade85539f4f9657 2023-04-06 17:58:06,093 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/60027ba21a48417c9ade85539f4f9657, entries=150, sequenceid=210, filesize=11.8 K 2023-04-06 17:58:06,093 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/48e5c7645d48465b977c39b9440fb704 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/48e5c7645d48465b977c39b9440fb704 2023-04-06 17:58:06,097 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/48e5c7645d48465b977c39b9440fb704, entries=150, sequenceid=210, filesize=11.8 K 2023-04-06 17:58:06,097 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~351.91 KB/360352, currentSize=53.67 KB/54960 for 6db618d61b9573465a7386d437c4d03f in 906ms, sequenceid=210, compaction requested=true 2023-04-06 17:58:06,097 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:06,097 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:06,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:06,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:06,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:06,098 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:06,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:06,099 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:06,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:06,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f4b556b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:06,102 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:06,102 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:06,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:06,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:06,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:06,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:06,107 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:06,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:06,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:06,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:06,107 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:06,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:06,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:06,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:06,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:06,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:06,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:06,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:06,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,123 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:06,123 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:06,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:06,224 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:06,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:06,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:06,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:06,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:06,231 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:06,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:06,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:06,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:06,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:06,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:06,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:06,233 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,233 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:06,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:06,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:06,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:06,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:06,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:06,235 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:06,235 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:06,235 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:06,235 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:06,235 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:06,235 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:06,235 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:06,235 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:58:06,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:06,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:06,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:06,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:06,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:06,236 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:06,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:06,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:06,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803946335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803886317, max timestamp=1680803886339Num uniques 859; 2023-04-06 17:58:06,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:06,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803886317, max timestamp=1680803886339Num uniques 859; 2023-04-06 17:58:06,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:06,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803886317, max timestamp=1680803886339Num uniques 859; 2023-04-06 17:58:06,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:06,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803946429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803946441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42528 deadline: 1680803946481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803946485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803946493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803946536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803946645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,654 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040695f6cb0715324d82ba6e44d4e7c8df72 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040695f6cb0715324d82ba6e44d4e7c8df72 2023-04-06 17:58:06,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803946741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:06,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:06,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803946946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803947048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,058 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/ab42fc6007554c78979949ee16b41443 2023-04-06 17:58:07,068 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/7ad05d905267413389b61057da19cfdf 2023-04-06 17:58:07,081 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/7207544a2c70425fb5a08313fc14c2cb 2023-04-06 17:58:07,087 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/ab42fc6007554c78979949ee16b41443 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/ab42fc6007554c78979949ee16b41443 2023-04-06 17:58:07,091 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/ab42fc6007554c78979949ee16b41443, entries=150, sequenceid=221, filesize=25.3 K 2023-04-06 17:58:07,092 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/7ad05d905267413389b61057da19cfdf as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7ad05d905267413389b61057da19cfdf 2023-04-06 17:58:07,095 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7ad05d905267413389b61057da19cfdf, entries=150, sequenceid=221, filesize=11.8 K 2023-04-06 17:58:07,096 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/7207544a2c70425fb5a08313fc14c2cb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7207544a2c70425fb5a08313fc14c2cb 2023-04-06 17:58:07,099 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7207544a2c70425fb5a08313fc14c2cb, entries=150, sequenceid=221, filesize=11.8 K 2023-04-06 17:58:07,099 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6db618d61b9573465a7386d437c4d03f in 864ms, sequenceid=221, compaction requested=true 2023-04-06 17:58:07,100 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:07,100 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:07,101 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:07,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:07,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:07,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:07,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:07,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:07,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:07,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7a2fd2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:07,103 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:07,103 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,103 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:07,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:07,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:07,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:07,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:07,104 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:07,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:07,104 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:07,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:07,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:07,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:07,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:07,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:07,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:07,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:07,106 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:07,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:07,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:07,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:07,107 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:07,107 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,107 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:07,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:07,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:07,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:07,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:07,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:07,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:07,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:07,111 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:07,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:07,111 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:07,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:07,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:07,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:07,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:07,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:07,211 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:07,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:07,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:07,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:07,216 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:07,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:07,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:07,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:07,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:07,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:07,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:07,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:07,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:07,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:07,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:07,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:07,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:07,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:07,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:07,222 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:07,222 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:07,222 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:07,222 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:07,222 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-06 17:58:07,222 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:07,222 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:07,222 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:07,223 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:07,223 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:07,223 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:07,257 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304060e256e9653b04c9d9ad6e9a6d7c88523 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060e256e9653b04c9d9ad6e9a6d7c88523 2023-04-06 17:58:07,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:07,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:07,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42528 deadline: 1680803947497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803947503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803947504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803947507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803947551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803947610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42514 deadline: 1680803947617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,670 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=257, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/c0130c101d804a7b890c3101b4226b19 2023-04-06 17:58:07,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803947815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:07,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:07,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803947822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,079 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/d720af1c2b8e46a6947006b3d8e9eebb 2023-04-06 17:58:08,100 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/57fe704c57204c168b3d2bf97f2faaa9 2023-04-06 17:58:08,103 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/c0130c101d804a7b890c3101b4226b19 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c0130c101d804a7b890c3101b4226b19 2023-04-06 17:58:08,106 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c0130c101d804a7b890c3101b4226b19, entries=150, sequenceid=257, filesize=25.3 K 2023-04-06 17:58:08,106 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/d720af1c2b8e46a6947006b3d8e9eebb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d720af1c2b8e46a6947006b3d8e9eebb 2023-04-06 17:58:08,109 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d720af1c2b8e46a6947006b3d8e9eebb, entries=150, sequenceid=257, filesize=11.8 K 2023-04-06 17:58:08,109 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/57fe704c57204c168b3d2bf97f2faaa9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/57fe704c57204c168b3d2bf97f2faaa9 2023-04-06 17:58:08,112 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/57fe704c57204c168b3d2bf97f2faaa9, entries=150, sequenceid=257, filesize=11.8 K 2023-04-06 17:58:08,112 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 6db618d61b9573465a7386d437c4d03f in 890ms, sequenceid=257, compaction requested=true 2023-04-06 17:58:08,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:08,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a68e7fc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,115 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,117 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,118 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:08,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:08,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,122 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:08,122 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:08,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:08,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:08,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803948203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803948204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,222 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:08,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:08,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:08,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:08,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:08,226 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:08,226 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,226 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:08,226 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:08,227 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,227 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,227 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,227 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,227 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,228 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:08,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:08,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:08,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:08,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:08,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:08,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:08,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:08,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:08,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,239 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bd14c8b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,241 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,241 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,242 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,247 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,247 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:08,248 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,248 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,248 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,250 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,251 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:08,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,251 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:08,251 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803948306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803948312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,352 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:08,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:08,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:08,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:08,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:08,354 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:08,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:08,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:08,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,356 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,356 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:08,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:08,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:08,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:08,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,358 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:08,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:08,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:08,361 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,362 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:08,362 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:08,362 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,363 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,364 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed4a297[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,364 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,365 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,365 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,366 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,366 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,366 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,366 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,369 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,370 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:08,370 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,370 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,370 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:08,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,377 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,377 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:08,377 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,477 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:08,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:08,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:08,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:08,483 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:08,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:08,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:08,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:08,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:08,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:08,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,487 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,487 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:08,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:08,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:08,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:08,490 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:08,490 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,490 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:08,491 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:08,491 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,498 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406e0620[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,500 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,500 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,501 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,503 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,503 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,503 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,503 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,503 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:08,504 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,504 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,504 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,504 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,508 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,508 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:08,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:08,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,509 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:08,509 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:08,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:08,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803948513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803948513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,538 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040617653e3bd1de44fea936925ccb16090f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040617653e3bd1de44fea936925ccb16090f 2023-04-06 17:58:08,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803948564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,609 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:08,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:08,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:08,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:08,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:08,613 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:08,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:08,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:08,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:08,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:08,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:08,616 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:08,616 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,616 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:08,616 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:08,616 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:08,617 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,617 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:08,617 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:08,617 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,617 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de45ed6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,620 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,620 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,620 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,621 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,621 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,621 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,626 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,626 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:08,631 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,631 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,632 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,632 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,633 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:08,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:08,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:08,635 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:08,636 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:08,736 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:08,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:08,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:08,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:08,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:08,739 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:08,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:08,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:08,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,741 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,741 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:08,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:08,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:08,742 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:08,742 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,742 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:08,743 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:08,743 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:08,743 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,743 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:08,743 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:08,743 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,743 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,743 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,743 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,743 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,743 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,744 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,744 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,745 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,745 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc0ecf5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,747 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,747 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,748 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,748 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,748 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,748 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,749 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,751 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,751 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,751 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,751 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,751 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:08,754 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,754 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,754 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,755 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,756 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:08,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,758 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:08,758 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:08,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:08,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803948817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803948817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,858 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:08,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:08,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:08,862 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:08,862 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,862 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:08,862 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:08,863 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,863 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,863 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,863 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:08,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:08,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:08,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:08,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:08,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:08,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:08,866 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,866 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:08,866 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:08,867 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@575b37e0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,871 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,871 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,872 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,872 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,875 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,875 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,875 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,875 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,875 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:08,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,881 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:08,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:08,881 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:08,881 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:08,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:08,942 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/bc6f8b7076f645528c76a9ef3e2239bc 2023-04-06 17:58:08,981 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:08,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:08,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:08,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:08,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:08,985 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:08,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:08,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:08,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:08,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:08,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:08,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:08,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:08,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:08,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:08,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:08,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:08,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:08,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:08,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:08,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6734642d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:08,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:08,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:08,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:08,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:08,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:08,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:08,995 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:08,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:08,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:08,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:08,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:08,999 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:08,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:08,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:08,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:08,999 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:09,002 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,002 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,002 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,005 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:09,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,007 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:09,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,007 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:09,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,107 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:09,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:09,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:09,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:09,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:09,110 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:09,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:09,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:09,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:09,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:09,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:09,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:09,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:09,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:09,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:09,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:09,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:09,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@589994f3[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:09,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:09,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,117 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:09,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:09,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:09,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:09,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:09,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:09,119 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:09,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:09,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:09,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:09,120 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:09,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,121 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:09,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:09,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:09,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,124 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:09,124 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:09,224 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:09,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:09,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:09,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:09,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:09,230 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:09,230 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,230 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:09,230 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:09,230 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,231 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,232 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:09,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:09,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:09,233 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:09,233 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,233 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:09,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:09,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:09,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:09,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:09,234 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:09,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:09,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:09,234 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:09,235 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,235 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:09,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53356c51[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:09,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:09,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,239 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:09,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:09,240 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:09,243 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:09,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:09,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:09,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:09,243 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:09,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,245 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,256 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:09,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:09,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:09,257 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,257 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:09,257 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:09,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803949327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42512 deadline: 1680803949328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/d06cea99a5904313bd247f4c9fbc3fa5 2023-04-06 17:58:09,357 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:09,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:09,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:09,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:09,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:09,374 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:09,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:09,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:09,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,377 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,378 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:09,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:09,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:09,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:09,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:09,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,379 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:09,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:09,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:09,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:09,381 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,381 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,382 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:09,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:09,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5fa5e0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:09,384 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:09,384 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,386 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:09,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:09,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:09,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,386 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:09,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:09,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:09,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:09,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:09,390 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:09,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:09,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:09,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:09,390 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:09,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,391 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,392 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,392 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,398 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,399 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:09,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:09,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:09,399 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:09,399 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:09,499 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:09,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:09,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:09,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:09,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:09,506 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:09,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:09,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:09,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,506 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:09,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:09,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:09,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:09,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:09,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:09,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:09,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:09,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:09,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:09,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42528 deadline: 1680803949518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,526 DEBUG [Thread-2310] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@198eb76a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:09,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:09,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:09,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:09,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:09,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:09,532 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:09,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:09,534 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:09,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:09,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:09,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:09,534 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:09,535 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,535 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,535 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803949528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,540 DEBUG [Thread-2312] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:09,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,549 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:09,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:09,550 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:09,550 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:09,650 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:09,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:09,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:09,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:09,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:09,657 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:09,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:09,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:09,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:09,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:09,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:09,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:09,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:09,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:09,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:09,670 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:09,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:09,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:09,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:09,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,683 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,683 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,683 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:09,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da496a4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:09,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:09,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,690 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:09,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:09,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:09,691 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,691 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:09,697 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,697 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:09,697 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,697 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:09,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:09,700 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:09,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:09,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:09,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:09,700 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:09,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,706 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,711 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:09,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:09,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:09,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,716 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:09,717 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:09,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:09,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:09,716 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:09,780 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/25eeb22af56047809f38f8c6e79f95d7 2023-04-06 17:58:09,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/bc6f8b7076f645528c76a9ef3e2239bc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/bc6f8b7076f645528c76a9ef3e2239bc 2023-04-06 17:58:09,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/bc6f8b7076f645528c76a9ef3e2239bc, entries=200, sequenceid=269, filesize=32.3 K 2023-04-06 17:58:09,787 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/d06cea99a5904313bd247f4c9fbc3fa5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d06cea99a5904313bd247f4c9fbc3fa5 2023-04-06 17:58:09,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d06cea99a5904313bd247f4c9fbc3fa5, entries=150, sequenceid=269, filesize=11.9 K 2023-04-06 17:58:09,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/25eeb22af56047809f38f8c6e79f95d7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/25eeb22af56047809f38f8c6e79f95d7 2023-04-06 17:58:09,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/25eeb22af56047809f38f8c6e79f95d7, entries=150, sequenceid=269, filesize=11.9 K 2023-04-06 17:58:09,802 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-06 17:58:09,804 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6db618d61b9573465a7386d437c4d03f in 1680ms, sequenceid=269, compaction requested=true 2023-04-06 17:58:09,804 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:09,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:09,804 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:58:09,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:09,804 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:58:09,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:09,812 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155978 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:58:09,812 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/A is initiating minor compaction (all files) 2023-04-06 17:58:09,813 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/A in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,813 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/faa479294b4341dc9839049c6c4093a1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/f9dbda42277746589395829912785b86, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/0de21d62611b443ea972eee8ce34ad9b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/ab42fc6007554c78979949ee16b41443, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c0130c101d804a7b890c3101b4226b19, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/bc6f8b7076f645528c76a9ef3e2239bc] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=152.3 K 2023-04-06 17:58:09,814 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70382 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:58:09,814 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/B is initiating minor compaction (all files) 2023-04-06 17:58:09,814 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/B in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,814 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/cc3f044c890742bfb4661ef3407d01e9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/3c071cec64544f81b3ac8c32c54e5674, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/60027ba21a48417c9ade85539f4f9657, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7ad05d905267413389b61057da19cfdf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d720af1c2b8e46a6947006b3d8e9eebb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d06cea99a5904313bd247f4c9fbc3fa5] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=68.7 K 2023-04-06 17:58:09,814 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting faa479294b4341dc9839049c6c4093a1, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803883634 2023-04-06 17:58:09,814 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f9dbda42277746589395829912785b86, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803883778 2023-04-06 17:58:09,815 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 0de21d62611b443ea972eee8ce34ad9b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680803884181 2023-04-06 17:58:09,815 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ab42fc6007554c78979949ee16b41443, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680803885336 2023-04-06 17:58:09,815 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c0130c101d804a7b890c3101b4226b19, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680803886411 2023-04-06 17:58:09,816 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting bc6f8b7076f645528c76a9ef3e2239bc, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680803887488 2023-04-06 17:58:09,816 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cc3f044c890742bfb4661ef3407d01e9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803883634 2023-04-06 17:58:09,816 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3c071cec64544f81b3ac8c32c54e5674, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803883778 2023-04-06 17:58:09,816 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 60027ba21a48417c9ade85539f4f9657, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680803884181 2023-04-06 17:58:09,817 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:09,817 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7ad05d905267413389b61057da19cfdf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680803885336 2023-04-06 17:58:09,817 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d720af1c2b8e46a6947006b3d8e9eebb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680803886411 2023-04-06 17:58:09,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:09,818 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d06cea99a5904313bd247f4c9fbc3fa5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680803887502 2023-04-06 17:58:09,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:09,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:09,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:09,830 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:09,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:09,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:09,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,831 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,832 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:09,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:09,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:09,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:09,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:09,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:09,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:09,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:09,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:09,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:09,837 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#B#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:09,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:09,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:09,837 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,837 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:09,838 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:58:09,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:09,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:09,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:09,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:09,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:09,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:09,843 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#A#compaction#358 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:09,879 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/43d722fb42ba42cfa0ac706cced6608f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/43d722fb42ba42cfa0ac706cced6608f 2023-04-06 17:58:09,883 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/B of 6db618d61b9573465a7386d437c4d03f into 43d722fb42ba42cfa0ac706cced6608f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:09,883 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:09,883 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/B, priority=10, startTime=1680803889804; duration=0sec 2023-04-06 17:58:09,883 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:09,883 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 17:58:09,884 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70382 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 17:58:09,884 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/C is initiating minor compaction (all files) 2023-04-06 17:58:09,884 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/C in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:09,885 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/750b487a3a624c6099d3e29c9a92c7cf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/cd9fb605048445369a3431742d48b39e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/48e5c7645d48465b977c39b9440fb704, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7207544a2c70425fb5a08313fc14c2cb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/57fe704c57204c168b3d2bf97f2faaa9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/25eeb22af56047809f38f8c6e79f95d7] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=68.7 K 2023-04-06 17:58:09,885 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 750b487a3a624c6099d3e29c9a92c7cf, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680803883634 2023-04-06 17:58:09,885 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cd9fb605048445369a3431742d48b39e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680803883778 2023-04-06 17:58:09,885 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 48e5c7645d48465b977c39b9440fb704, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680803884181 2023-04-06 17:58:09,886 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7207544a2c70425fb5a08313fc14c2cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680803885336 2023-04-06 17:58:09,886 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 57fe704c57204c168b3d2bf97f2faaa9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680803886411 2023-04-06 17:58:09,887 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 25eeb22af56047809f38f8c6e79f95d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680803887502 2023-04-06 17:58:09,906 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#C#compaction#361 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:09,907 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b46aa53a6f494620acad0f3d8b1ad034 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b46aa53a6f494620acad0f3d8b1ad034 2023-04-06 17:58:09,938 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/8f2d2a1e9a1e47d69d37d2530ae76290 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/8f2d2a1e9a1e47d69d37d2530ae76290 2023-04-06 17:58:09,943 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/C of 6db618d61b9573465a7386d437c4d03f into 8f2d2a1e9a1e47d69d37d2530ae76290(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:09,943 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:09,943 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/C, priority=10, startTime=1680803889804; duration=0sec 2023-04-06 17:58:09,943 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:10,336 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/6d72b97446bc4f899131b1ebba02ef45 2023-04-06 17:58:10,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:10,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:10,354 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/7207387195394b02acfccd5ea636c274 2023-04-06 17:58:10,365 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/3f0964436bd24bfd969e3fa668f8fd57 2023-04-06 17:58:10,369 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/6d72b97446bc4f899131b1ebba02ef45 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6d72b97446bc4f899131b1ebba02ef45 2023-04-06 17:58:10,375 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6d72b97446bc4f899131b1ebba02ef45, entries=150, sequenceid=289, filesize=25.4 K 2023-04-06 17:58:10,376 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/7207387195394b02acfccd5ea636c274 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7207387195394b02acfccd5ea636c274 2023-04-06 17:58:10,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803950375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,380 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7207387195394b02acfccd5ea636c274, entries=150, sequenceid=289, filesize=11.9 K 2023-04-06 17:58:10,380 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/3f0964436bd24bfd969e3fa668f8fd57 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3f0964436bd24bfd969e3fa668f8fd57 2023-04-06 17:58:10,384 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3f0964436bd24bfd969e3fa668f8fd57, entries=150, sequenceid=289, filesize=11.9 K 2023-04-06 17:58:10,384 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6db618d61b9573465a7386d437c4d03f in 547ms, sequenceid=289, compaction requested=false 2023-04-06 17:58:10,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:10,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:10,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:10,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:10,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:10,386 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@758b34a5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:10,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:10,389 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,390 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:10,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:10,390 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:10,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,390 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:10,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:10,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:58:10,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:10,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:10,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:10,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:10,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:10,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:10,395 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:10,396 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,396 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:10,395 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:10,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:10,398 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:10,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:10,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:10,398 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:10,398 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:10,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,400 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,404 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,407 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,408 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:10,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:10,408 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:10,408 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:10,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:10,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803950497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803950497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,508 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:10,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:10,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:10,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:10,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:10,513 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:10,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:10,513 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:10,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,514 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,515 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,515 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:10,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:10,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:10,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:10,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:10,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:10,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:10,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:10,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:10,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:10,518 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:10,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:10,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:10,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:10,520 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:10,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:10,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555bff51[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:10,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:10,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,526 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:10,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:10,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:10,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:10,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:10,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:10,532 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:10,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:10,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:10,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:10,532 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:10,550 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,555 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,556 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,558 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,558 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:10,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,559 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,559 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:10,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803950586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,591 DEBUG [Thread-2314] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:10,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803950604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803950605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,659 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:10,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:10,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:10,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:10,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:10,662 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:10,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:10,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:10,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,664 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,664 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:10,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:10,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:10,665 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,665 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:10,665 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:10,665 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:10,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:10,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:10,666 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,666 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:10,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,666 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:10,670 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:10,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39822777[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:10,670 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,671 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:10,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:10,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:10,671 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,671 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:10,672 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:10,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:10,674 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:10,674 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:10,674 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:10,674 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:10,674 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:10,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,675 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,676 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,678 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,679 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:10,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,679 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:10,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,679 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,679 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:10,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:10,708 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/872cd6b9e58445ffb4462cb39016d1ff as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/872cd6b9e58445ffb4462cb39016d1ff 2023-04-06 17:58:10,712 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/A of 6db618d61b9573465a7386d437c4d03f into 872cd6b9e58445ffb4462cb39016d1ff(size=26.0 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:10,712 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:10,713 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/A, priority=10, startTime=1680803889804; duration=0sec 2023-04-06 17:58:10,713 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:10,779 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:10,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:10,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:10,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:10,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:10,782 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:10,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:10,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:10,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,784 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,784 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:10,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:10,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:10,785 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,785 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:10,785 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:10,785 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:10,785 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:10,786 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,786 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,787 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,787 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:10,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103a9755[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:10,791 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:10,791 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,792 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:10,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:10,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:10,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:10,793 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:10,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:10,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:10,797 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:10,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:10,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:10,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:10,797 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:10,798 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,798 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,798 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,801 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,802 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:10,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:10,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,806 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:10,806 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:10,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803950809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:10,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803950813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,847 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406967fd6e4d499434cb74cf375c4f17381 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406967fd6e4d499434cb74cf375c4f17381 2023-04-06 17:58:10,860 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/15522c5484994fefb741eb254308a8b3 2023-04-06 17:58:10,907 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:10,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:10,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:10,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:10,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:10,909 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:10,909 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,909 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:10,909 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:10,909 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,909 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,910 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,910 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,910 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,910 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:10,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:10,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:10,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:10,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:10,911 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:10,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:10,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:10,912 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,912 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:10,912 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:10,912 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:10,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:10,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:10,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:10,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:10,913 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:10,914 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:10,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb8a7eb[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:10,914 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,915 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:10,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:10,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:10,915 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:10,916 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:10,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,916 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,916 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:10,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:10,917 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:10,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:10,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:10,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:10,918 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:10,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:10,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:10,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,920 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:10,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:10,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:10,921 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:10,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:10,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:10,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:10,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:10,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:10,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:10,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:10,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:10,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:10,922 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,922 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:10,922 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:10,922 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:10,922 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,022 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:11,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:11,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:11,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:11,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:11,025 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:11,025 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,025 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:11,025 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:11,025 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,025 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,026 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:11,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,026 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:11,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:11,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:11,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:11,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:11,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:11,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:11,027 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:11,027 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,028 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:11,028 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:11,028 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:11,028 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,029 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:11,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3693665a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:11,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:11,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,030 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,030 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:11,030 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:11,030 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,031 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:11,031 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:11,031 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:11,031 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:11,031 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:11,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:11,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:11,033 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:11,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:11,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:11,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:11,033 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:11,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:11,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:11,035 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,036 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,048 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,048 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,048 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:11,048 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:11,048 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:11,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:11,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:11,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:11,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:11,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:11,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:11,049 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:11,049 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:11,048 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,050 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803951114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803951121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,149 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:11,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:11,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:11,152 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:11,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:11,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:11,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,153 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:11,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:11,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:11,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:11,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:11,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:11,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:11,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:11,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:11,154 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,154 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:11,154 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:11,154 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:11,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:11,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:11,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:11,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,155 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,156 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,156 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:11,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e824660[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:11,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:11,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:11,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:11,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:11,157 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:11,158 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:11,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:11,158 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:11,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:11,160 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:11,160 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:11,160 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:11,160 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:11,160 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:11,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:11,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,161 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,162 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,164 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:11,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:11,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:11,165 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:11,165 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:11,165 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:11,165 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:11,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:11,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:11,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:11,265 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:11,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:11,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:11,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:11,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:11,268 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:11,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:11,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:11,269 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,269 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:11,269 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:11,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:11,270 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,270 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:11,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:11,272 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/769a4374abfb46219cb4313418584923 2023-04-06 17:58:11,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:11,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:11,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:11,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:11,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:11,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:11,272 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,272 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:11,272 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:11,272 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:11,273 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:11,273 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,273 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:11,273 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:11,274 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,274 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,274 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,274 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:11,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ac6eada[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:11,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:11,278 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:11,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:11,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:11,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:11,280 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,280 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:11,280 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:11,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,280 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:11,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:11,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:11,288 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:11,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:11,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:11,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:11,288 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:11,290 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,290 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:11,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:11,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,300 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:11,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:11,300 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:11,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:11,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:11,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:11,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:11,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:11,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:11,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:11,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:11,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:11,306 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:11,306 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:11,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/35a4c9ea1ef84639ab0b3658a5f8365c 2023-04-06 17:58:11,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/15522c5484994fefb741eb254308a8b3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/15522c5484994fefb741eb254308a8b3 2023-04-06 17:58:11,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/15522c5484994fefb741eb254308a8b3, entries=200, sequenceid=304, filesize=32.3 K 2023-04-06 17:58:11,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/769a4374abfb46219cb4313418584923 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/769a4374abfb46219cb4313418584923 2023-04-06 17:58:11,359 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/769a4374abfb46219cb4313418584923, entries=150, sequenceid=304, filesize=11.9 K 2023-04-06 17:58:11,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/35a4c9ea1ef84639ab0b3658a5f8365c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/35a4c9ea1ef84639ab0b3658a5f8365c 2023-04-06 17:58:11,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/35a4c9ea1ef84639ab0b3658a5f8365c, entries=150, sequenceid=304, filesize=11.9 K 2023-04-06 17:58:11,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6db618d61b9573465a7386d437c4d03f in 972ms, sequenceid=304, compaction requested=true 2023-04-06 17:58:11,364 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:11,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:11,364 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:11,364 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:11,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:11,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:11,367 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:11,367 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/A is initiating minor compaction (all files) 2023-04-06 17:58:11,367 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/A in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,367 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/872cd6b9e58445ffb4462cb39016d1ff, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6d72b97446bc4f899131b1ebba02ef45, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/15522c5484994fefb741eb254308a8b3] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=83.8 K 2023-04-06 17:58:11,368 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:11,368 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/B is initiating minor compaction (all files) 2023-04-06 17:58:11,368 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 872cd6b9e58445ffb4462cb39016d1ff, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680803887502 2023-04-06 17:58:11,368 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/B in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,368 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/43d722fb42ba42cfa0ac706cced6608f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7207387195394b02acfccd5ea636c274, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/769a4374abfb46219cb4313418584923] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=36.3 K 2023-04-06 17:58:11,374 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 43d722fb42ba42cfa0ac706cced6608f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680803887502 2023-04-06 17:58:11,374 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6d72b97446bc4f899131b1ebba02ef45, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680803888192 2023-04-06 17:58:11,375 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7207387195394b02acfccd5ea636c274, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680803888192 2023-04-06 17:58:11,375 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 15522c5484994fefb741eb254308a8b3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680803890357 2023-04-06 17:58:11,376 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 769a4374abfb46219cb4313418584923, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680803890374 2023-04-06 17:58:11,397 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#B#compaction#368 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:11,405 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#A#compaction#367 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:11,406 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:11,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:11,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:11,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:11,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:11,429 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:11,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:11,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:11,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:11,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:11,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:11,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:11,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:11,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:11,441 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:11,442 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:11,442 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:11,442 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:11,460 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040658512426971f4166b4e59180bf8fba9f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040658512426971f4166b4e59180bf8fba9f 2023-04-06 17:58:11,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:11,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:11,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803951649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803951658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803951758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803951763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,821 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/6567f19dba0c4e44942fcb63ce14ebc2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/6567f19dba0c4e44942fcb63ce14ebc2 2023-04-06 17:58:11,825 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/B of 6db618d61b9573465a7386d437c4d03f into 6567f19dba0c4e44942fcb63ce14ebc2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:11,825 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:11,825 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/B, priority=13, startTime=1680803891364; duration=0sec 2023-04-06 17:58:11,825 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:11,825 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:11,826 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:11,826 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/C is initiating minor compaction (all files) 2023-04-06 17:58:11,826 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/C in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:11,826 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/8f2d2a1e9a1e47d69d37d2530ae76290, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3f0964436bd24bfd969e3fa668f8fd57, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/35a4c9ea1ef84639ab0b3658a5f8365c] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=36.3 K 2023-04-06 17:58:11,826 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 8f2d2a1e9a1e47d69d37d2530ae76290, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680803887502 2023-04-06 17:58:11,827 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3f0964436bd24bfd969e3fa668f8fd57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680803888192 2023-04-06 17:58:11,827 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 35a4c9ea1ef84639ab0b3658a5f8365c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680803890374 2023-04-06 17:58:11,832 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#C#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:11,878 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=324, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/6dab279e879c4c9ebc9bd62fddd8d38a 2023-04-06 17:58:11,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803951965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:11,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:11,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803951966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,244 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/e4a5234c23ee423cb24767dee0350dd1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e4a5234c23ee423cb24767dee0350dd1 2023-04-06 17:58:12,249 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/C of 6db618d61b9573465a7386d437c4d03f into e4a5234c23ee423cb24767dee0350dd1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:12,249 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:12,249 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/C, priority=13, startTime=1680803891365; duration=0sec 2023-04-06 17:58:12,250 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:12,256 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/464e9d77cbdd4518bbe3965277807ec7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/464e9d77cbdd4518bbe3965277807ec7 2023-04-06 17:58:12,266 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/A of 6db618d61b9573465a7386d437c4d03f into 464e9d77cbdd4518bbe3965277807ec7(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:12,266 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:12,266 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/A, priority=13, startTime=1680803891364; duration=0sec 2023-04-06 17:58:12,266 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:12,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:12,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803952273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:12,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803952274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,290 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/2732aa5e777b48db97f237f3774700eb 2023-04-06 17:58:12,298 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/5e3573c111474d94981207479e10693b 2023-04-06 17:58:12,301 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/6dab279e879c4c9ebc9bd62fddd8d38a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6dab279e879c4c9ebc9bd62fddd8d38a 2023-04-06 17:58:12,304 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6dab279e879c4c9ebc9bd62fddd8d38a, entries=150, sequenceid=324, filesize=25.4 K 2023-04-06 17:58:12,304 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/2732aa5e777b48db97f237f3774700eb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/2732aa5e777b48db97f237f3774700eb 2023-04-06 17:58:12,307 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/2732aa5e777b48db97f237f3774700eb, entries=150, sequenceid=324, filesize=11.9 K 2023-04-06 17:58:12,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/5e3573c111474d94981207479e10693b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/5e3573c111474d94981207479e10693b 2023-04-06 17:58:12,310 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/5e3573c111474d94981207479e10693b, entries=150, sequenceid=324, filesize=11.9 K 2023-04-06 17:58:12,311 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6db618d61b9573465a7386d437c4d03f in 869ms, sequenceid=324, compaction requested=false 2023-04-06 17:58:12,311 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:12,311 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:12,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:12,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:12,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:12,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:12,312 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:12,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,312 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:12,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:12,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23389996[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:12,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:12,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:12,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:12,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:12,316 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,316 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:12,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:12,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:12,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:12,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:12,318 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:12,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:12,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:12,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:12,319 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:12,319 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,319 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:12,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,319 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:12,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:12,320 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,322 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:12,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:12,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:12,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:12,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:12,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:12,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:12,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:12,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,324 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:12,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,324 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:12,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:12,424 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:12,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:12,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:12,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:12,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:12,428 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:12,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:12,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:12,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:12,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:12,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:12,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:12,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:12,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:12,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:12,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:12,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:12,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:12,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:12,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:12,442 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:12,442 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:12,452 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:12,452 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:12,452 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:58:12,453 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:12,453 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:12,453 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:12,453 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:12,453 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:12,453 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:12,479 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406275f2a62c14642c8a92043864f31ab66 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406275f2a62c14642c8a92043864f31ab66 2023-04-06 17:58:12,488 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=340, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/fa9eab3d9e034081a374c8c35640d117 2023-04-06 17:58:12,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:12,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:12,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:12,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803952830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:12,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803952838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,900 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/ddf41623e94846c0acbb514ad7f924e4 2023-04-06 17:58:12,926 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/7f509f5e9ea046a79364b5fd24c75451 2023-04-06 17:58:12,929 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/fa9eab3d9e034081a374c8c35640d117 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/fa9eab3d9e034081a374c8c35640d117 2023-04-06 17:58:12,944 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/fa9eab3d9e034081a374c8c35640d117, entries=150, sequenceid=340, filesize=25.4 K 2023-04-06 17:58:12,944 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/ddf41623e94846c0acbb514ad7f924e4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ddf41623e94846c0acbb514ad7f924e4 2023-04-06 17:58:12,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:12,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803952939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,948 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ddf41623e94846c0acbb514ad7f924e4, entries=150, sequenceid=340, filesize=11.9 K 2023-04-06 17:58:12,948 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/7f509f5e9ea046a79364b5fd24c75451 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7f509f5e9ea046a79364b5fd24c75451 2023-04-06 17:58:12,951 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7f509f5e9ea046a79364b5fd24c75451, entries=150, sequenceid=340, filesize=11.9 K 2023-04-06 17:58:12,952 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6db618d61b9573465a7386d437c4d03f in 500ms, sequenceid=340, compaction requested=true 2023-04-06 17:58:12,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:12,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:12,953 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:12,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:12,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:12,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e348a1d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:12,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:12,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,955 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:12,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:12,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:12,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:12,958 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:12,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:12,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:12,958 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:12,958 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:12,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,959 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:12,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:12,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:12,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,964 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:12,964 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:12,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:12,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:12,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:12,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:12,964 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:12,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:12,964 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:12,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:12,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:12,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:12,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:12,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:12,966 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803953028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,065 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:13,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:13,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:13,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:13,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:13,067 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:13,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:13,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:13,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,069 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,069 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,070 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:13,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:13,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:13,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:13,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,070 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:13,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:13,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:13,071 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:13,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,072 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:13,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:13,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2096cfd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:13,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,073 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:13,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,073 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:13,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:13,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:13,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:13,079 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:13,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:13,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:13,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:13,079 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:13,080 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:13,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:13,083 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:13,083 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:13,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803953140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803953145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,183 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:13,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:13,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:13,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:13,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:13,193 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:13,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:13,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:13,194 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,194 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,194 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,194 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,195 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,195 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:13,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:13,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:13,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,207 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:13,210 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:13,210 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:13,210 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,210 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:13,210 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:13,210 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,210 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:13,210 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:13,210 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,210 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:13,210 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:13,211 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,211 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,211 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,211 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:13,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff4c65c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:13,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:13,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,213 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:13,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,213 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,214 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:13,216 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:13,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:13,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:13,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:13,217 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:13,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,218 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,219 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,221 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:13,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,221 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:13,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,221 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:13,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:13,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,321 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:13,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:13,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:13,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:13,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:13,324 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:13,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:13,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:13,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:13,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:13,326 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,326 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:13,326 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:13,326 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:13,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,327 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,327 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,327 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:13,328 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:13,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e7161d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:13,328 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,329 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,329 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:13,329 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:13,329 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:13,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,329 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:13,329 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,329 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:13,331 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:13,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:13,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:13,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:13,331 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:13,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,333 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,336 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,336 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,336 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:13,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:13,336 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,337 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:13,337 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:13,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:13,336 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,337 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,337 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,337 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,338 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,338 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803953350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,393 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304066904d103ca204a7dab92f92500faccfb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066904d103ca204a7dab92f92500faccfb 2023-04-06 17:58:13,399 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/5f89ffcfc2d3418aaff2e69c62947219 2023-04-06 17:58:13,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/e87f4ed4aef1407da48c58174264c380 2023-04-06 17:58:13,437 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:13,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:13,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:13,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:13,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:13,440 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:13,440 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,440 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:13,440 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:13,441 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,441 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,441 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,442 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,442 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:13,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:13,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:13,443 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,443 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:13,449 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:13,449 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:13,449 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:13,449 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:13,458 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:13,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a9017e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:13,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:13,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803953453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,464 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,465 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:13,465 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:13,465 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:13,465 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:13,466 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,467 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:13,467 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,467 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:13,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:13,468 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:13,468 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:13,468 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:13,468 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:13,468 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:13,469 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,469 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,469 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,470 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:13,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:13,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,471 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,472 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,472 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:13,472 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,472 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,472 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42528 deadline: 1680803953544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,551 DEBUG [Thread-2310] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:13,572 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:13,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:13,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:13,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:13,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:13,575 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:13,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:13,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:13,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,576 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,576 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,576 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,576 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:13,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:13,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:13,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:13,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:13,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:13,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:13,577 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,577 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:13,577 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:13,577 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:13,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:13,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:13,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:13,578 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,579 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,579 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:13,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b8eff52[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:13,580 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:13,580 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,580 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42588 deadline: 1680803953576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:13,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:13,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:13,581 DEBUG [Thread-2312] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8221 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:13,581 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,582 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:13,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,582 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,582 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,582 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:13,584 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:13,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:13,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:13,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:13,584 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:13,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,585 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,587 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:13,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:13,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,589 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,590 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,590 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,590 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:13,590 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:13,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:13,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803953657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,690 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:13,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:13,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:13,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:13,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:13,694 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:13,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:13,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:13,695 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,695 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,696 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:13,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:13,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:13,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:13,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:13,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:13,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:13,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:13,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:13,714 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,714 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:13,714 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:13,714 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:13,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:13,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:13,715 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:13,715 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,716 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,716 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,716 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:13,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22810259[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:13,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:13,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,728 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:13,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:13,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:13,734 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:13,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:13,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:13,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:13,734 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:13,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:13,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:13,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,746 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:13,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:13,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,749 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:13,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,749 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:13,749 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:13,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:13,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/b8b22762bb5b449fa4b286a23ad86876 2023-04-06 17:58:13,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/5f89ffcfc2d3418aaff2e69c62947219 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5f89ffcfc2d3418aaff2e69c62947219 2023-04-06 17:58:13,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5f89ffcfc2d3418aaff2e69c62947219, entries=200, sequenceid=360, filesize=32.3 K 2023-04-06 17:58:13,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/e87f4ed4aef1407da48c58174264c380 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/e87f4ed4aef1407da48c58174264c380 2023-04-06 17:58:13,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/e87f4ed4aef1407da48c58174264c380, entries=150, sequenceid=360, filesize=11.9 K 2023-04-06 17:58:13,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/b8b22762bb5b449fa4b286a23ad86876 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/b8b22762bb5b449fa4b286a23ad86876 2023-04-06 17:58:13,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/b8b22762bb5b449fa4b286a23ad86876, entries=150, sequenceid=360, filesize=11.9 K 2023-04-06 17:58:13,833 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6db618d61b9573465a7386d437c4d03f in 872ms, sequenceid=360, compaction requested=true 2023-04-06 17:58:13,833 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:13,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:13,834 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:13,834 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:13,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:13,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:13,835 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111940 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:13,835 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/A is initiating minor compaction (all files) 2023-04-06 17:58:13,835 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/A in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,835 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/464e9d77cbdd4518bbe3965277807ec7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6dab279e879c4c9ebc9bd62fddd8d38a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/fa9eab3d9e034081a374c8c35640d117, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5f89ffcfc2d3418aaff2e69c62947219] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=109.3 K 2023-04-06 17:58:13,835 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:13,835 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/B is initiating minor compaction (all files) 2023-04-06 17:58:13,835 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 464e9d77cbdd4518bbe3965277807ec7, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680803890374 2023-04-06 17:58:13,835 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/B in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,835 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/6567f19dba0c4e44942fcb63ce14ebc2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/2732aa5e777b48db97f237f3774700eb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ddf41623e94846c0acbb514ad7f924e4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/e87f4ed4aef1407da48c58174264c380] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=48.3 K 2023-04-06 17:58:13,835 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6dab279e879c4c9ebc9bd62fddd8d38a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680803890487 2023-04-06 17:58:13,836 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6567f19dba0c4e44942fcb63ce14ebc2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680803890374 2023-04-06 17:58:13,836 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting fa9eab3d9e034081a374c8c35640d117, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680803891642 2023-04-06 17:58:13,836 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2732aa5e777b48db97f237f3774700eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680803890487 2023-04-06 17:58:13,836 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 5f89ffcfc2d3418aaff2e69c62947219, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680803892822 2023-04-06 17:58:13,836 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting ddf41623e94846c0acbb514ad7f924e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680803891642 2023-04-06 17:58:13,836 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e87f4ed4aef1407da48c58174264c380, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680803892822 2023-04-06 17:58:13,850 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:13,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:13,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:13,863 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#B#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:13,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:13,865 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:13,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:13,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:13,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,866 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:13,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:13,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:13,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:13,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:13,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:13,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:13,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:13,877 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:13,877 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:13,877 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:13,877 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:13,877 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:13,877 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,877 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:13,877 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:58:13,878 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:13,878 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#A#compaction#379 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:13,880 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:13,880 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:13,880 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:13,880 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:13,880 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:13,911 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/aa44a384252d4451829cb726f4cf2ac0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/aa44a384252d4451829cb726f4cf2ac0 2023-04-06 17:58:13,926 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/B of 6db618d61b9573465a7386d437c4d03f into aa44a384252d4451829cb726f4cf2ac0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:13,926 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:13,926 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/B, priority=12, startTime=1680803893834; duration=0sec 2023-04-06 17:58:13,926 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:13,926 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:13,927 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:13,928 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/C is initiating minor compaction (all files) 2023-04-06 17:58:13,928 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/C in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:13,928 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e4a5234c23ee423cb24767dee0350dd1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/5e3573c111474d94981207479e10693b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7f509f5e9ea046a79364b5fd24c75451, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/b8b22762bb5b449fa4b286a23ad86876] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=48.3 K 2023-04-06 17:58:13,928 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e4a5234c23ee423cb24767dee0350dd1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680803890374 2023-04-06 17:58:13,928 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5e3573c111474d94981207479e10693b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680803890487 2023-04-06 17:58:13,929 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7f509f5e9ea046a79364b5fd24c75451, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680803891642 2023-04-06 17:58:13,930 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b8b22762bb5b449fa4b286a23ad86876, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680803892822 2023-04-06 17:58:13,956 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#C#compaction#382 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:13,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:13,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:13,983 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/3c2eeb3a94244dfbbd83f179f319e5f6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3c2eeb3a94244dfbbd83f179f319e5f6 2023-04-06 17:58:13,990 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/C of 6db618d61b9573465a7386d437c4d03f into 3c2eeb3a94244dfbbd83f179f319e5f6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:13,990 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:13,990 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/C, priority=12, startTime=1680803893834; duration=0sec 2023-04-06 17:58:13,990 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:14,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:14,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803954099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:14,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803954163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:14,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803954209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,359 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406306d91022280405bb66a9c036052e67b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406306d91022280405bb66a9c036052e67b 2023-04-06 17:58:14,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:14,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803954416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:14,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42572 deadline: 1680803954622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,631 DEBUG [Thread-2314] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:14,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:14,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803954726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,798 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/259262d3b5834329b1b612d647cd9bd4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/259262d3b5834329b1b612d647cd9bd4 2023-04-06 17:58:14,804 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/A of 6db618d61b9573465a7386d437c4d03f into 259262d3b5834329b1b612d647cd9bd4(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:14,804 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:14,804 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/A, priority=12, startTime=1680803893833; duration=0sec 2023-04-06 17:58:14,804 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:14,808 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/a41fe5b7911541ec839b558e602aa365 2023-04-06 17:58:14,833 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/c3a923b30f3441b6b4f0ff7c7236752c 2023-04-06 17:58:14,847 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/c91aa71e8c344cff9a6e403a5d674f4d 2023-04-06 17:58:14,852 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/a41fe5b7911541ec839b558e602aa365 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/a41fe5b7911541ec839b558e602aa365 2023-04-06 17:58:14,856 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/a41fe5b7911541ec839b558e602aa365, entries=150, sequenceid=372, filesize=25.4 K 2023-04-06 17:58:14,857 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/c3a923b30f3441b6b4f0ff7c7236752c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c3a923b30f3441b6b4f0ff7c7236752c 2023-04-06 17:58:14,861 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c3a923b30f3441b6b4f0ff7c7236752c, entries=150, sequenceid=372, filesize=11.9 K 2023-04-06 17:58:14,862 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/c91aa71e8c344cff9a6e403a5d674f4d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c91aa71e8c344cff9a6e403a5d674f4d 2023-04-06 17:58:14,866 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c91aa71e8c344cff9a6e403a5d674f4d, entries=150, sequenceid=372, filesize=11.9 K 2023-04-06 17:58:14,867 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6db618d61b9573465a7386d437c4d03f in 990ms, sequenceid=372, compaction requested=false 2023-04-06 17:58:14,867 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:14,867 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:14,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:14,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:14,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:14,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:14,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:14,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:14,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:14,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:14,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1266afea[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:14,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,870 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:14,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:14,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,871 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:14,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:14,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:14,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:14,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:14,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:14,874 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:14,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:14,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:14,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:14,874 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:14,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:14,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:14,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:14,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:14,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:14,878 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:14,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:14,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:14,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:14,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:14,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:14,880 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:14,880 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:14,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:14,980 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:14,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:14,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:14,985 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:14,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:14,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:14,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:14,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:14,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:14,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:14,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:14,990 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:14,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:15,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:15,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:15,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:15,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:15,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:15,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:15,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:15,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:15,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:15,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:15,012 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:15,012 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. started... 2023-04-06 17:58:15,012 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:58:15,013 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:15,013 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:15,013 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:15,013 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:15,013 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:15,013 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:15,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:15,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. as already flushing 2023-04-06 17:58:15,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803955241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803955248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,279 DEBUG [Thread-2323] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1bd0918a to 127.0.0.1:49384 2023-04-06 17:58:15,279 DEBUG [Thread-2323] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:15,281 DEBUG [Thread-2321] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cb8db43 to 127.0.0.1:49384 2023-04-06 17:58:15,281 DEBUG [Thread-2321] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:15,282 DEBUG [Thread-2327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x144047c5 to 127.0.0.1:49384 2023-04-06 17:58:15,282 DEBUG [Thread-2327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:15,287 DEBUG [Thread-2325] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5225f8f6 to 127.0.0.1:49384 2023-04-06 17:58:15,287 DEBUG [Thread-2325] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:15,292 DEBUG [Thread-2329] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x525c59d1 to 127.0.0.1:49384 2023-04-06 17:58:15,292 DEBUG [Thread-2329] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:15,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803955354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803955356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,491 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304066462e37b5f474b75a5223ebb0b73bad2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066462e37b5f474b75a5223ebb0b73bad2 2023-04-06 17:58:15,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803955557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803955557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803955859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:15,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803955859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:15,904 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/392bd976cf7144ed84be905d059d77d7 2023-04-06 17:58:16,313 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/f937258522db4b73bbb9c167a7fa0195 2023-04-06 17:58:16,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:16,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42512 deadline: 1680803956361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:16,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42514 deadline: 1680803956363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,723 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/87ae003255d04945b7bf5887e9483120 2023-04-06 17:58:16,730 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/392bd976cf7144ed84be905d059d77d7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/392bd976cf7144ed84be905d059d77d7 2023-04-06 17:58:16,736 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/392bd976cf7144ed84be905d059d77d7, entries=150, sequenceid=395, filesize=25.4 K 2023-04-06 17:58:16,736 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/f937258522db4b73bbb9c167a7fa0195 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f937258522db4b73bbb9c167a7fa0195 2023-04-06 17:58:16,743 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f937258522db4b73bbb9c167a7fa0195, entries=150, sequenceid=395, filesize=11.9 K 2023-04-06 17:58:16,744 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/87ae003255d04945b7bf5887e9483120 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/87ae003255d04945b7bf5887e9483120 2023-04-06 17:58:16,749 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/87ae003255d04945b7bf5887e9483120, entries=150, sequenceid=395, filesize=11.9 K 2023-04-06 17:58:16,750 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6db618d61b9573465a7386d437c4d03f in 1738ms, sequenceid=395, compaction requested=true 2023-04-06 17:58:16,750 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:16,750 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:16,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:16,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:16,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:16,750 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:16,752 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:16,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@782eee29[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:16,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:16,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:16,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:16,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:16,756 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,756 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:16,757 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:16,757 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:16,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,757 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:16,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:16,760 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:16,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:16,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:16,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:16,761 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:16,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:16,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:16,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:16,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:16,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:16,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:16,767 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:16,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:16,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:16,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:16,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:16,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:16,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:16,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:16,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:16,780 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:16,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:16,781 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:16,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:16,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:16,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:16,881 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:16,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:17,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:17,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:58:17,367 DEBUG [Thread-2316] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f29a773 to 127.0.0.1:49384 2023-04-06 17:58:17,367 DEBUG [Thread-2316] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:17,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:17,368 DEBUG [Thread-2318] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48681e8f to 127.0.0.1:49384 2023-04-06 17:58:17,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:17,368 DEBUG [Thread-2318] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:17,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:17,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:17,369 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:17,369 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:17,781 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040629c1e59ffea140369dfb57ba0dbfe5a2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040629c1e59ffea140369dfb57ba0dbfe5a2 2023-04-06 17:58:18,189 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=409, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d41810662539482cac462e5a505e1e75 2023-04-06 17:58:18,234 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-06 17:58:18,265 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d8c1f6c5bc99482a89565d507bc72d9c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/10c8b74d0e0142dea7c137dc77e7c294, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5b443c757ba1422fb92a37604645ab7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b42f32c751d440979e9edc11857a2b17, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d7f4ae8691914936aa8662318b96b95a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d4344136d307420688a8520e2ba6854d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/abc619725b6344cba66e2924c1548340, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/1c483b6da7784fb29c7e6eca59372202, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b4331337ac0b472ead6849d82b8d57d5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/82498514a0e740ed84049609480666d4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b5d3ae88d9ec40d0ba99f7cff18bb291, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c1dd342ba6a34d748e2c83970d7573d8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/faa479294b4341dc9839049c6c4093a1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/f9dbda42277746589395829912785b86, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/0de21d62611b443ea972eee8ce34ad9b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/ab42fc6007554c78979949ee16b41443, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c0130c101d804a7b890c3101b4226b19, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/bc6f8b7076f645528c76a9ef3e2239bc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/872cd6b9e58445ffb4462cb39016d1ff, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6d72b97446bc4f899131b1ebba02ef45, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/15522c5484994fefb741eb254308a8b3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/464e9d77cbdd4518bbe3965277807ec7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6dab279e879c4c9ebc9bd62fddd8d38a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/fa9eab3d9e034081a374c8c35640d117, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5f89ffcfc2d3418aaff2e69c62947219] to archive 2023-04-06 17:58:18,266 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:18,267 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d8c1f6c5bc99482a89565d507bc72d9c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d8c1f6c5bc99482a89565d507bc72d9c 2023-04-06 17:58:18,268 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/10c8b74d0e0142dea7c137dc77e7c294 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/10c8b74d0e0142dea7c137dc77e7c294 2023-04-06 17:58:18,269 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5b443c757ba1422fb92a37604645ab7a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5b443c757ba1422fb92a37604645ab7a 2023-04-06 17:58:18,270 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b42f32c751d440979e9edc11857a2b17 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b42f32c751d440979e9edc11857a2b17 2023-04-06 17:58:18,271 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d7f4ae8691914936aa8662318b96b95a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d7f4ae8691914936aa8662318b96b95a 2023-04-06 17:58:18,272 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d4344136d307420688a8520e2ba6854d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d4344136d307420688a8520e2ba6854d 2023-04-06 17:58:18,273 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/abc619725b6344cba66e2924c1548340 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/abc619725b6344cba66e2924c1548340 2023-04-06 17:58:18,274 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/1c483b6da7784fb29c7e6eca59372202 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/1c483b6da7784fb29c7e6eca59372202 2023-04-06 17:58:18,275 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b4331337ac0b472ead6849d82b8d57d5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b4331337ac0b472ead6849d82b8d57d5 2023-04-06 17:58:18,275 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/82498514a0e740ed84049609480666d4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/82498514a0e740ed84049609480666d4 2023-04-06 17:58:18,277 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b5d3ae88d9ec40d0ba99f7cff18bb291 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/b5d3ae88d9ec40d0ba99f7cff18bb291 2023-04-06 17:58:18,278 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c1dd342ba6a34d748e2c83970d7573d8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c1dd342ba6a34d748e2c83970d7573d8 2023-04-06 17:58:18,279 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/faa479294b4341dc9839049c6c4093a1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/faa479294b4341dc9839049c6c4093a1 2023-04-06 17:58:18,280 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/f9dbda42277746589395829912785b86 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/f9dbda42277746589395829912785b86 2023-04-06 17:58:18,281 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/0de21d62611b443ea972eee8ce34ad9b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/0de21d62611b443ea972eee8ce34ad9b 2023-04-06 17:58:18,283 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/ab42fc6007554c78979949ee16b41443 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/ab42fc6007554c78979949ee16b41443 2023-04-06 17:58:18,284 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c0130c101d804a7b890c3101b4226b19 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/c0130c101d804a7b890c3101b4226b19 2023-04-06 17:58:18,285 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/bc6f8b7076f645528c76a9ef3e2239bc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/bc6f8b7076f645528c76a9ef3e2239bc 2023-04-06 17:58:18,286 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/872cd6b9e58445ffb4462cb39016d1ff to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/872cd6b9e58445ffb4462cb39016d1ff 2023-04-06 17:58:18,288 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6d72b97446bc4f899131b1ebba02ef45 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6d72b97446bc4f899131b1ebba02ef45 2023-04-06 17:58:18,289 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/15522c5484994fefb741eb254308a8b3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/15522c5484994fefb741eb254308a8b3 2023-04-06 17:58:18,290 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/464e9d77cbdd4518bbe3965277807ec7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/464e9d77cbdd4518bbe3965277807ec7 2023-04-06 17:58:18,291 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6dab279e879c4c9ebc9bd62fddd8d38a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/6dab279e879c4c9ebc9bd62fddd8d38a 2023-04-06 17:58:18,293 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/fa9eab3d9e034081a374c8c35640d117 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/fa9eab3d9e034081a374c8c35640d117 2023-04-06 17:58:18,294 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5f89ffcfc2d3418aaff2e69c62947219 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/5f89ffcfc2d3418aaff2e69c62947219 2023-04-06 17:58:18,301 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/25628d6a8fbd421b84c2fe306d2a1afe, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a2e3cc86cfe6437eaaf1f2da8645a211, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/4b60ebd1e867459987112cef15bc161b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a8422fc40f524383adf0dd1997029b93, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/acb5f02abc9d449baca069b6639812fc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c6d9b1d9ce1e44d5b94eb8ff6b002049, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ae768ba8cfe7473db7d8779f5d9814e0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/93d1d0386f90451ba315db386d58a10f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c36dc6d806d74bd19f254601335e4017, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a5dfbe3de5f0489788aca9d6312d15cc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f2e22fc438794a948bcf7b54be1d7d2b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/cc3f044c890742bfb4661ef3407d01e9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/b44e6fa5483c40b5b51905e75d199a7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/3c071cec64544f81b3ac8c32c54e5674, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/60027ba21a48417c9ade85539f4f9657, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7ad05d905267413389b61057da19cfdf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d720af1c2b8e46a6947006b3d8e9eebb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/43d722fb42ba42cfa0ac706cced6608f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d06cea99a5904313bd247f4c9fbc3fa5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7207387195394b02acfccd5ea636c274, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/6567f19dba0c4e44942fcb63ce14ebc2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/769a4374abfb46219cb4313418584923, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/2732aa5e777b48db97f237f3774700eb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ddf41623e94846c0acbb514ad7f924e4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/e87f4ed4aef1407da48c58174264c380] to archive 2023-04-06 17:58:18,302 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:18,303 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/25628d6a8fbd421b84c2fe306d2a1afe to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/25628d6a8fbd421b84c2fe306d2a1afe 2023-04-06 17:58:18,304 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a2e3cc86cfe6437eaaf1f2da8645a211 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a2e3cc86cfe6437eaaf1f2da8645a211 2023-04-06 17:58:18,305 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/4b60ebd1e867459987112cef15bc161b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/4b60ebd1e867459987112cef15bc161b 2023-04-06 17:58:18,305 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a8422fc40f524383adf0dd1997029b93 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a8422fc40f524383adf0dd1997029b93 2023-04-06 17:58:18,306 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/acb5f02abc9d449baca069b6639812fc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/acb5f02abc9d449baca069b6639812fc 2023-04-06 17:58:18,307 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c6d9b1d9ce1e44d5b94eb8ff6b002049 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c6d9b1d9ce1e44d5b94eb8ff6b002049 2023-04-06 17:58:18,308 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ae768ba8cfe7473db7d8779f5d9814e0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ae768ba8cfe7473db7d8779f5d9814e0 2023-04-06 17:58:18,308 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/93d1d0386f90451ba315db386d58a10f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/93d1d0386f90451ba315db386d58a10f 2023-04-06 17:58:18,309 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c36dc6d806d74bd19f254601335e4017 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c36dc6d806d74bd19f254601335e4017 2023-04-06 17:58:18,310 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a5dfbe3de5f0489788aca9d6312d15cc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/a5dfbe3de5f0489788aca9d6312d15cc 2023-04-06 17:58:18,310 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f2e22fc438794a948bcf7b54be1d7d2b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f2e22fc438794a948bcf7b54be1d7d2b 2023-04-06 17:58:18,311 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/cc3f044c890742bfb4661ef3407d01e9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/cc3f044c890742bfb4661ef3407d01e9 2023-04-06 17:58:18,312 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/b44e6fa5483c40b5b51905e75d199a7a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/b44e6fa5483c40b5b51905e75d199a7a 2023-04-06 17:58:18,312 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/3c071cec64544f81b3ac8c32c54e5674 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/3c071cec64544f81b3ac8c32c54e5674 2023-04-06 17:58:18,313 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/60027ba21a48417c9ade85539f4f9657 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/60027ba21a48417c9ade85539f4f9657 2023-04-06 17:58:18,314 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7ad05d905267413389b61057da19cfdf to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7ad05d905267413389b61057da19cfdf 2023-04-06 17:58:18,315 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d720af1c2b8e46a6947006b3d8e9eebb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d720af1c2b8e46a6947006b3d8e9eebb 2023-04-06 17:58:18,316 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/43d722fb42ba42cfa0ac706cced6608f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/43d722fb42ba42cfa0ac706cced6608f 2023-04-06 17:58:18,318 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d06cea99a5904313bd247f4c9fbc3fa5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/d06cea99a5904313bd247f4c9fbc3fa5 2023-04-06 17:58:18,319 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7207387195394b02acfccd5ea636c274 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/7207387195394b02acfccd5ea636c274 2023-04-06 17:58:18,320 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/6567f19dba0c4e44942fcb63ce14ebc2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/6567f19dba0c4e44942fcb63ce14ebc2 2023-04-06 17:58:18,321 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/769a4374abfb46219cb4313418584923 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/769a4374abfb46219cb4313418584923 2023-04-06 17:58:18,322 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/2732aa5e777b48db97f237f3774700eb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/2732aa5e777b48db97f237f3774700eb 2023-04-06 17:58:18,323 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ddf41623e94846c0acbb514ad7f924e4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/ddf41623e94846c0acbb514ad7f924e4 2023-04-06 17:58:18,325 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/e87f4ed4aef1407da48c58174264c380 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/e87f4ed4aef1407da48c58174264c380 2023-04-06 17:58:18,331 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/bc8eb6490cf64a5eba0b07f48cba3387, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/82282f9e44d74a54b1e666e1c5a6cade, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/576795bdff16445aba827c1cb2c28316, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/052ca744eaa4417c91cc59e689ca9201, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/9323121315734ec0b25eada158b4cec6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/83858d2ef09b494aa9a7d561b097e50b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/55cdfc8656e84e7bacd82961a7a35dbd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/f100261c31dd4da2bc9c7e5ac33a281d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e05184ac796f4c85b89706ba793b4312, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/a2dc8800ebe74c17811feaf5a254b7d5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/99f1048629ac47c7be9091624105831e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/750b487a3a624c6099d3e29c9a92c7cf, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c3c212775e98489b939312213be3d287, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/cd9fb605048445369a3431742d48b39e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/48e5c7645d48465b977c39b9440fb704, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7207544a2c70425fb5a08313fc14c2cb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/57fe704c57204c168b3d2bf97f2faaa9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/8f2d2a1e9a1e47d69d37d2530ae76290, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/25eeb22af56047809f38f8c6e79f95d7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3f0964436bd24bfd969e3fa668f8fd57, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e4a5234c23ee423cb24767dee0350dd1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/35a4c9ea1ef84639ab0b3658a5f8365c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/5e3573c111474d94981207479e10693b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7f509f5e9ea046a79364b5fd24c75451, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/b8b22762bb5b449fa4b286a23ad86876] to archive 2023-04-06 17:58:18,332 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:18,334 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/bc8eb6490cf64a5eba0b07f48cba3387 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/bc8eb6490cf64a5eba0b07f48cba3387 2023-04-06 17:58:18,335 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/82282f9e44d74a54b1e666e1c5a6cade to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/82282f9e44d74a54b1e666e1c5a6cade 2023-04-06 17:58:18,336 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/576795bdff16445aba827c1cb2c28316 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/576795bdff16445aba827c1cb2c28316 2023-04-06 17:58:18,336 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/052ca744eaa4417c91cc59e689ca9201 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/052ca744eaa4417c91cc59e689ca9201 2023-04-06 17:58:18,337 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/9323121315734ec0b25eada158b4cec6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/9323121315734ec0b25eada158b4cec6 2023-04-06 17:58:18,338 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/83858d2ef09b494aa9a7d561b097e50b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/83858d2ef09b494aa9a7d561b097e50b 2023-04-06 17:58:18,339 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/55cdfc8656e84e7bacd82961a7a35dbd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/55cdfc8656e84e7bacd82961a7a35dbd 2023-04-06 17:58:18,340 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/f100261c31dd4da2bc9c7e5ac33a281d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/f100261c31dd4da2bc9c7e5ac33a281d 2023-04-06 17:58:18,341 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e05184ac796f4c85b89706ba793b4312 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e05184ac796f4c85b89706ba793b4312 2023-04-06 17:58:18,342 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/a2dc8800ebe74c17811feaf5a254b7d5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/a2dc8800ebe74c17811feaf5a254b7d5 2023-04-06 17:58:18,343 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/99f1048629ac47c7be9091624105831e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/99f1048629ac47c7be9091624105831e 2023-04-06 17:58:18,344 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/750b487a3a624c6099d3e29c9a92c7cf to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/750b487a3a624c6099d3e29c9a92c7cf 2023-04-06 17:58:18,345 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c3c212775e98489b939312213be3d287 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c3c212775e98489b939312213be3d287 2023-04-06 17:58:18,346 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/cd9fb605048445369a3431742d48b39e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/cd9fb605048445369a3431742d48b39e 2023-04-06 17:58:18,346 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/48e5c7645d48465b977c39b9440fb704 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/48e5c7645d48465b977c39b9440fb704 2023-04-06 17:58:18,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7207544a2c70425fb5a08313fc14c2cb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7207544a2c70425fb5a08313fc14c2cb 2023-04-06 17:58:18,348 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/57fe704c57204c168b3d2bf97f2faaa9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/57fe704c57204c168b3d2bf97f2faaa9 2023-04-06 17:58:18,348 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/8f2d2a1e9a1e47d69d37d2530ae76290 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/8f2d2a1e9a1e47d69d37d2530ae76290 2023-04-06 17:58:18,349 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/25eeb22af56047809f38f8c6e79f95d7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/25eeb22af56047809f38f8c6e79f95d7 2023-04-06 17:58:18,350 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3f0964436bd24bfd969e3fa668f8fd57 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3f0964436bd24bfd969e3fa668f8fd57 2023-04-06 17:58:18,350 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e4a5234c23ee423cb24767dee0350dd1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/e4a5234c23ee423cb24767dee0350dd1 2023-04-06 17:58:18,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/35a4c9ea1ef84639ab0b3658a5f8365c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/35a4c9ea1ef84639ab0b3658a5f8365c 2023-04-06 17:58:18,352 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/5e3573c111474d94981207479e10693b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/5e3573c111474d94981207479e10693b 2023-04-06 17:58:18,352 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7f509f5e9ea046a79364b5fd24c75451 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/7f509f5e9ea046a79364b5fd24c75451 2023-04-06 17:58:18,353 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/b8b22762bb5b449fa4b286a23ad86876 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/b8b22762bb5b449fa4b286a23ad86876 2023-04-06 17:58:18,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/fd9084a676324b798dee1b02041007b3 2023-04-06 17:58:19,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/749bc53354f748c4bb7ec5e48b582bff 2023-04-06 17:58:19,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/d41810662539482cac462e5a505e1e75 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d41810662539482cac462e5a505e1e75 2023-04-06 17:58:19,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d41810662539482cac462e5a505e1e75, entries=150, sequenceid=409, filesize=25.4 K 2023-04-06 17:58:19,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/fd9084a676324b798dee1b02041007b3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/fd9084a676324b798dee1b02041007b3 2023-04-06 17:58:19,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/fd9084a676324b798dee1b02041007b3, entries=150, sequenceid=409, filesize=11.9 K 2023-04-06 17:58:19,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/749bc53354f748c4bb7ec5e48b582bff as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/749bc53354f748c4bb7ec5e48b582bff 2023-04-06 17:58:19,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/749bc53354f748c4bb7ec5e48b582bff, entries=150, sequenceid=409, filesize=11.9 K 2023-04-06 17:58:19,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 6db618d61b9573465a7386d437c4d03f in 1673ms, sequenceid=409, compaction requested=true 2023-04-06 17:58:19,040 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:19,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:19,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:19,040 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:19,041 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:19,041 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:19,042 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:19,042 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:19,042 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/A is initiating minor compaction (all files) 2023-04-06 17:58:19,042 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/B is initiating minor compaction (all files) 2023-04-06 17:58:19,043 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/A in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:19,043 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/B in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:19,043 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/259262d3b5834329b1b612d647cd9bd4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/a41fe5b7911541ec839b558e602aa365, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/392bd976cf7144ed84be905d059d77d7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d41810662539482cac462e5a505e1e75] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=102.6 K 2023-04-06 17:58:19,043 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/aa44a384252d4451829cb726f4cf2ac0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c3a923b30f3441b6b4f0ff7c7236752c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f937258522db4b73bbb9c167a7fa0195, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/fd9084a676324b798dee1b02041007b3] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=48.4 K 2023-04-06 17:58:19,043 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 259262d3b5834329b1b612d647cd9bd4, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680803892822 2023-04-06 17:58:19,043 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting aa44a384252d4451829cb726f4cf2ac0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680803892822 2023-04-06 17:58:19,044 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a41fe5b7911541ec839b558e602aa365, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680803892999 2023-04-06 17:58:19,044 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c3a923b30f3441b6b4f0ff7c7236752c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680803892999 2023-04-06 17:58:19,044 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 392bd976cf7144ed84be905d059d77d7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680803894077 2023-04-06 17:58:19,044 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f937258522db4b73bbb9c167a7fa0195, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680803894077 2023-04-06 17:58:19,044 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d41810662539482cac462e5a505e1e75, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1680803895212 2023-04-06 17:58:19,044 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting fd9084a676324b798dee1b02041007b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1680803895212 2023-04-06 17:58:19,054 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#B#compaction#391 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:19,056 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#A#compaction#392 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:19,463 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/c07ef52981d1416bbce34533c0330d73 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c07ef52981d1416bbce34533c0330d73 2023-04-06 17:58:19,469 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/B of 6db618d61b9573465a7386d437c4d03f into c07ef52981d1416bbce34533c0330d73(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:19,469 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:19,469 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/B, priority=12, startTime=1680803899040; duration=0sec 2023-04-06 17:58:19,470 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:19,470 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:19,472 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:19,472 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 6db618d61b9573465a7386d437c4d03f/C is initiating minor compaction (all files) 2023-04-06 17:58:19,472 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6db618d61b9573465a7386d437c4d03f/C in TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:19,472 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3c2eeb3a94244dfbbd83f179f319e5f6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c91aa71e8c344cff9a6e403a5d674f4d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/87ae003255d04945b7bf5887e9483120, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/749bc53354f748c4bb7ec5e48b582bff] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp, totalSize=48.4 K 2023-04-06 17:58:19,472 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3c2eeb3a94244dfbbd83f179f319e5f6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680803892822 2023-04-06 17:58:19,473 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c91aa71e8c344cff9a6e403a5d674f4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680803892999 2023-04-06 17:58:19,473 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 87ae003255d04945b7bf5887e9483120, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680803894077 2023-04-06 17:58:19,474 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 749bc53354f748c4bb7ec5e48b582bff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1680803895212 2023-04-06 17:58:19,484 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6db618d61b9573465a7386d437c4d03f#C#compaction#393 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:19,897 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/948addf1ee8a4cf1bc0f8978c0662347 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/948addf1ee8a4cf1bc0f8978c0662347 2023-04-06 17:58:19,905 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/C of 6db618d61b9573465a7386d437c4d03f into 948addf1ee8a4cf1bc0f8978c0662347(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:19,905 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:19,905 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/C, priority=12, startTime=1680803899041; duration=0sec 2023-04-06 17:58:19,905 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:20,286 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/85d01800d2064dc0827e3ca5eb0ed49e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/85d01800d2064dc0827e3ca5eb0ed49e 2023-04-06 17:58:20,292 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6db618d61b9573465a7386d437c4d03f/A of 6db618d61b9573465a7386d437c4d03f into 85d01800d2064dc0827e3ca5eb0ed49e(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 17:58:20,292 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:20,293 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f., storeName=6db618d61b9573465a7386d437c4d03f/A, priority=12, startTime=1680803899040; duration=1sec 2023-04-06 17:58:20,293 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:23,617 DEBUG [Thread-2310] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a8219c3 to 127.0.0.1:49384 2023-04-06 17:58:23,617 DEBUG [Thread-2310] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:23,664 DEBUG [Thread-2312] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ed5d233 to 127.0.0.1:49384 2023-04-06 17:58:23,664 DEBUG [Thread-2312] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:24,674 DEBUG [Thread-2314] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a409627 to 127.0.0.1:49384 2023-04-06 17:58:24,674 DEBUG [Thread-2314] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:24,674 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 17:58:24,674 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 1816 2023-04-06 17:58:24,675 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 5448 rows 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 1819 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 5457 rows 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 1843 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 5529 rows 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 1858 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 5574 rows 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 1843 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 5529 rows 2023-04-06 17:58:24,676 INFO [Listener at localhost.localdomain/40169] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 17:58:24,677 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20312fe6 to 127.0.0.1:49384 2023-04-06 17:58:24,677 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:24,680 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 17:58:24,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 17:58:24,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:24,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 17:58:24,685 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803904685"}]},"ts":"1680803904685"} 2023-04-06 17:58:24,686 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 17:58:24,705 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 17:58:24,705 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, UNASSIGN}] 2023-04-06 17:58:24,706 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, UNASSIGN 2023-04-06 17:58:24,707 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:24,707 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803904707"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803904707"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803904707"}]},"ts":"1680803904707"} 2023-04-06 17:58:24,708 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:58:24,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 17:58:24,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:24,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 6db618d61b9573465a7386d437c4d03f, disabling compactions & flushes 2023-04-06 17:58:24,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:24,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:24,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. after waiting 0 ms 2023-04-06 17:58:24,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:24,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 6db618d61b9573465a7386d437c4d03f 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 17:58:24,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=A 2023-04-06 17:58:24,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:24,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=B 2023-04-06 17:58:24,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:24,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6db618d61b9573465a7386d437c4d03f, store=C 2023-04-06 17:58:24,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:24,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 17:58:25,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406a680d6382aef4591baa67ef3217d0030 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a680d6382aef4591baa67ef3217d0030 2023-04-06 17:58:25,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 17:58:25,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=418, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/43274663c40f44af98e791a921726dfd 2023-04-06 17:58:25,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 17:58:26,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/319380d349b74127822e0df914b485b3 2023-04-06 17:58:26,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/2aebd3def82c4534990d746ff74694a3 2023-04-06 17:58:26,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/A/43274663c40f44af98e791a921726dfd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/43274663c40f44af98e791a921726dfd 2023-04-06 17:58:26,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/43274663c40f44af98e791a921726dfd, entries=50, sequenceid=418, filesize=11.7 K 2023-04-06 17:58:26,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/B/319380d349b74127822e0df914b485b3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/319380d349b74127822e0df914b485b3 2023-04-06 17:58:26,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/319380d349b74127822e0df914b485b3, entries=50, sequenceid=418, filesize=7.1 K 2023-04-06 17:58:26,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/.tmp/C/2aebd3def82c4534990d746ff74694a3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/2aebd3def82c4534990d746ff74694a3 2023-04-06 17:58:26,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/2aebd3def82c4534990d746ff74694a3, entries=50, sequenceid=418, filesize=7.1 K 2023-04-06 17:58:26,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 6db618d61b9573465a7386d437c4d03f in 1690ms, sequenceid=418, compaction requested=false 2023-04-06 17:58:26,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/259262d3b5834329b1b612d647cd9bd4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/a41fe5b7911541ec839b558e602aa365, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/392bd976cf7144ed84be905d059d77d7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d41810662539482cac462e5a505e1e75] to archive 2023-04-06 17:58:26,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:26,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/259262d3b5834329b1b612d647cd9bd4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/259262d3b5834329b1b612d647cd9bd4 2023-04-06 17:58:26,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/a41fe5b7911541ec839b558e602aa365 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/a41fe5b7911541ec839b558e602aa365 2023-04-06 17:58:26,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/392bd976cf7144ed84be905d059d77d7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/392bd976cf7144ed84be905d059d77d7 2023-04-06 17:58:26,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d41810662539482cac462e5a505e1e75 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/d41810662539482cac462e5a505e1e75 2023-04-06 17:58:26,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/aa44a384252d4451829cb726f4cf2ac0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c3a923b30f3441b6b4f0ff7c7236752c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f937258522db4b73bbb9c167a7fa0195, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/fd9084a676324b798dee1b02041007b3] to archive 2023-04-06 17:58:26,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:26,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/aa44a384252d4451829cb726f4cf2ac0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/aa44a384252d4451829cb726f4cf2ac0 2023-04-06 17:58:26,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c3a923b30f3441b6b4f0ff7c7236752c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c3a923b30f3441b6b4f0ff7c7236752c 2023-04-06 17:58:26,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f937258522db4b73bbb9c167a7fa0195 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/f937258522db4b73bbb9c167a7fa0195 2023-04-06 17:58:26,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/fd9084a676324b798dee1b02041007b3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/fd9084a676324b798dee1b02041007b3 2023-04-06 17:58:26,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3c2eeb3a94244dfbbd83f179f319e5f6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c91aa71e8c344cff9a6e403a5d674f4d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/87ae003255d04945b7bf5887e9483120, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/749bc53354f748c4bb7ec5e48b582bff] to archive 2023-04-06 17:58:26,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:26,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3c2eeb3a94244dfbbd83f179f319e5f6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/3c2eeb3a94244dfbbd83f179f319e5f6 2023-04-06 17:58:26,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c91aa71e8c344cff9a6e403a5d674f4d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/c91aa71e8c344cff9a6e403a5d674f4d 2023-04-06 17:58:26,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/87ae003255d04945b7bf5887e9483120 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/87ae003255d04945b7bf5887e9483120 2023-04-06 17:58:26,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/749bc53354f748c4bb7ec5e48b582bff to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/749bc53354f748c4bb7ec5e48b582bff 2023-04-06 17:58:26,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/recovered.edits/421.seqid, newMaxSeqId=421, maxSeqId=4 2023-04-06 17:58:26,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f. 2023-04-06 17:58:26,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 6db618d61b9573465a7386d437c4d03f: 2023-04-06 17:58:26,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:26,592 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=6db618d61b9573465a7386d437c4d03f, regionState=CLOSED 2023-04-06 17:58:26,593 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803906592"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803906592"}]},"ts":"1680803906592"} 2023-04-06 17:58:26,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-06 17:58:26,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 6db618d61b9573465a7386d437c4d03f, server=jenkins-hbase20.apache.org,38773,1680803776439 in 1.8860 sec 2023-04-06 17:58:26,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-06 17:58:26,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6db618d61b9573465a7386d437c4d03f, UNASSIGN in 1.8910 sec 2023-04-06 17:58:26,597 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803906597"}]},"ts":"1680803906597"} 2023-04-06 17:58:26,598 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 17:58:26,599 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 17:58:26,601 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9190 sec 2023-04-06 17:58:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 17:58:26,790 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-06 17:58:26,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 17:58:26,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:26,793 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:26,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-06 17:58:26,794 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:26,798 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:26,800 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/recovered.edits] 2023-04-06 17:58:26,803 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/43274663c40f44af98e791a921726dfd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/43274663c40f44af98e791a921726dfd 2023-04-06 17:58:26,804 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/85d01800d2064dc0827e3ca5eb0ed49e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/A/85d01800d2064dc0827e3ca5eb0ed49e 2023-04-06 17:58:26,806 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/319380d349b74127822e0df914b485b3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/319380d349b74127822e0df914b485b3 2023-04-06 17:58:26,807 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c07ef52981d1416bbce34533c0330d73 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/B/c07ef52981d1416bbce34533c0330d73 2023-04-06 17:58:26,809 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/2aebd3def82c4534990d746ff74694a3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/2aebd3def82c4534990d746ff74694a3 2023-04-06 17:58:26,810 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/948addf1ee8a4cf1bc0f8978c0662347 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/C/948addf1ee8a4cf1bc0f8978c0662347 2023-04-06 17:58:26,812 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/recovered.edits/421.seqid to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f/recovered.edits/421.seqid 2023-04-06 17:58:26,813 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/6db618d61b9573465a7386d437c4d03f 2023-04-06 17:58:26,813 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:58:26,813 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 17:58:26,814 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 17:58:26,816 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406013d04fcfdac48ebbef9a3777a25986b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406013d04fcfdac48ebbef9a3777a25986b 2023-04-06 17:58:26,817 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060a8b28bd807c4efa8f343c8565014e87 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060a8b28bd807c4efa8f343c8565014e87 2023-04-06 17:58:26,818 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060e256e9653b04c9d9ad6e9a6d7c88523 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060e256e9653b04c9d9ad6e9a6d7c88523 2023-04-06 17:58:26,819 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040617653e3bd1de44fea936925ccb16090f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040617653e3bd1de44fea936925ccb16090f 2023-04-06 17:58:26,819 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061f3bd3bca00341a0b513b4735879886d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061f3bd3bca00341a0b513b4735879886d 2023-04-06 17:58:26,820 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406275f2a62c14642c8a92043864f31ab66 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406275f2a62c14642c8a92043864f31ab66 2023-04-06 17:58:26,821 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040629c1e59ffea140369dfb57ba0dbfe5a2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040629c1e59ffea140369dfb57ba0dbfe5a2 2023-04-06 17:58:26,821 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f88e50ff6e545ba83c3e096c727664a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f88e50ff6e545ba83c3e096c727664a 2023-04-06 17:58:26,822 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406306d91022280405bb66a9c036052e67b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406306d91022280405bb66a9c036052e67b 2023-04-06 17:58:26,823 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040636df98ccc3404a0c93040b809a456c70 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040636df98ccc3404a0c93040b809a456c70 2023-04-06 17:58:26,823 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406395fa00e3ef947549923d11f9ebc8e24 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406395fa00e3ef947549923d11f9ebc8e24 2023-04-06 17:58:26,824 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040658512426971f4166b4e59180bf8fba9f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040658512426971f4166b4e59180bf8fba9f 2023-04-06 17:58:26,825 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066462e37b5f474b75a5223ebb0b73bad2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066462e37b5f474b75a5223ebb0b73bad2 2023-04-06 17:58:26,825 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066904d103ca204a7dab92f92500faccfb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066904d103ca204a7dab92f92500faccfb 2023-04-06 17:58:26,826 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040681b05793cc8b4de3b3410e16bd4b19c5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040681b05793cc8b4de3b3410e16bd4b19c5 2023-04-06 17:58:26,827 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040695f6cb0715324d82ba6e44d4e7c8df72 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040695f6cb0715324d82ba6e44d4e7c8df72 2023-04-06 17:58:26,827 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406967fd6e4d499434cb74cf375c4f17381 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406967fd6e4d499434cb74cf375c4f17381 2023-04-06 17:58:26,828 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a5682899683a4567b0aa14191aa91a21 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a5682899683a4567b0aa14191aa91a21 2023-04-06 17:58:26,829 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a680d6382aef4591baa67ef3217d0030 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a680d6382aef4591baa67ef3217d0030 2023-04-06 17:58:26,830 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b27bffa6d2684ca6a99b0feece8bebc3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b27bffa6d2684ca6a99b0feece8bebc3 2023-04-06 17:58:26,831 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b46aa53a6f494620acad0f3d8b1ad034 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b46aa53a6f494620acad0f3d8b1ad034 2023-04-06 17:58:26,831 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c6b95749dbbb4632a0c903f1add0505e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c6b95749dbbb4632a0c903f1add0505e 2023-04-06 17:58:26,832 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d99b9d4a331f4f9494e71b9a60a43bd6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d99b9d4a331f4f9494e71b9a60a43bd6 2023-04-06 17:58:26,833 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ff0742276f4d465a9ca2d699f941d6bb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ff0742276f4d465a9ca2d699f941d6bb 2023-04-06 17:58:26,833 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 17:58:26,835 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:26,836 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 17:58:26,837 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 17:58:26,838 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:26,838 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 17:58:26,838 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680803906838"}]},"ts":"9223372036854775807"} 2023-04-06 17:58:26,839 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 17:58:26,839 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6db618d61b9573465a7386d437c4d03f, NAME => 'TestAcidGuarantees,,1680803872175.6db618d61b9573465a7386d437c4d03f.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 17:58:26,839 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 17:58:26,839 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680803906839"}]},"ts":"9223372036854775807"} 2023-04-06 17:58:26,840 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 17:58:26,841 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:26,842 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 50 msec 2023-04-06 17:58:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-06 17:58:26,895 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-06 17:58:26,906 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 227), OpenFileDescriptor=512 (was 544), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=417 (was 450), ProcessCount=167 (was 166) - ProcessCount LEAK? -, AvailableMemoryMB=3751 (was 4405) 2023-04-06 17:58:26,915 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=512, MaxFileDescriptor=60000, SystemLoadAverage=417, ProcessCount=167, AvailableMemoryMB=3751 2023-04-06 17:58:26,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:58:26,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:58:26,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:26,920 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 17:58:26,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-06 17:58:26,920 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 17:58:26,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 17:58:26,921 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:26,922 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d empty. 2023-04-06 17:58:26,922 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:26,922 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:58:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 17:58:27,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 17:58:27,331 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:58:27,335 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ec62d27e82d7188e5d149482edb0608d, NAME => 'TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp 2023-04-06 17:58:27,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 17:58:27,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:58:27,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ec62d27e82d7188e5d149482edb0608d, disabling compactions & flushes 2023-04-06 17:58:27,744 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:27,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:27,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. after waiting 0 ms 2023-04-06 17:58:27,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:27,744 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:27,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:27,747 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 17:58:27,749 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803907749"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803907749"}]},"ts":"1680803907749"} 2023-04-06 17:58:27,750 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 17:58:27,751 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 17:58:27,751 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803907751"}]},"ts":"1680803907751"} 2023-04-06 17:58:27,753 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 17:58:27,756 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec62d27e82d7188e5d149482edb0608d, ASSIGN}] 2023-04-06 17:58:27,758 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec62d27e82d7188e5d149482edb0608d, ASSIGN 2023-04-06 17:58:27,759 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec62d27e82d7188e5d149482edb0608d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=false 2023-04-06 17:58:27,910 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=ec62d27e82d7188e5d149482edb0608d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:27,911 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803907910"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803907910"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803907910"}]},"ts":"1680803907910"} 2023-04-06 17:58:27,912 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:58:28,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 17:58:28,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:28,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ec62d27e82d7188e5d149482edb0608d, NAME => 'TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:58:28,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:58:28,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,074 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,077 DEBUG [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A 2023-04-06 17:58:28,077 DEBUG [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A 2023-04-06 17:58:28,077 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:58:28,077 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec62d27e82d7188e5d149482edb0608d columnFamilyName A 2023-04-06 17:58:28,078 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.HStore(310): Store=ec62d27e82d7188e5d149482edb0608d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:58:28,078 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,081 DEBUG [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B 2023-04-06 17:58:28,081 DEBUG [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B 2023-04-06 17:58:28,081 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:58:28,081 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec62d27e82d7188e5d149482edb0608d columnFamilyName B 2023-04-06 17:58:28,082 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.HStore(310): Store=ec62d27e82d7188e5d149482edb0608d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:58:28,082 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,084 DEBUG [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C 2023-04-06 17:58:28,084 DEBUG [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C 2023-04-06 17:58:28,085 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:58:28,085 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec62d27e82d7188e5d149482edb0608d columnFamilyName C 2023-04-06 17:58:28,085 INFO [StoreOpener-ec62d27e82d7188e5d149482edb0608d-1] regionserver.HStore(310): Store=ec62d27e82d7188e5d149482edb0608d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:58:28,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:28,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:58:28,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:28,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:58:28,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ec62d27e82d7188e5d149482edb0608d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74826726, jitterRate=0.11500510573387146}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:58:28,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:28,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., pid=49, masterSystemTime=1680803908064 2023-04-06 17:58:28,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:28,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:28,096 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=ec62d27e82d7188e5d149482edb0608d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:28,096 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803908096"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803908096"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803908096"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803908096"}]},"ts":"1680803908096"} 2023-04-06 17:58:28,100 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-06 17:58:28,100 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 in 185 msec 2023-04-06 17:58:28,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-06 17:58:28,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec62d27e82d7188e5d149482edb0608d, ASSIGN in 344 msec 2023-04-06 17:58:28,103 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 17:58:28,103 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803908103"}]},"ts":"1680803908103"} 2023-04-06 17:58:28,104 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 17:58:28,107 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 17:58:28,108 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1890 sec 2023-04-06 17:58:29,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 17:58:29,030 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-06 17:58:29,032 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x796f86e5 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,041 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5534caaa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,042 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,044 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,045 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 17:58:29,047 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 17:58:29,048 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x4391d6f4 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,057 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46cc28cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,057 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x17a38ae8 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,065 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a48c89e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,066 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x3e21fa39 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,074 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@355ed3df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,074 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x7cb49278 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,082 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4df55fe5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,083 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x50e89dd2 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,090 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@817e2e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,090 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x4048dcb4 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,096 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@777f1e92, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,097 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x4e7a5e9a to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,107 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48d1969d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,107 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x62447900 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,113 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed6755c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,114 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x519db5f2 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,124 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10505ab8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,124 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x5c9c1feb to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:29,132 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@609213af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:29,137 DEBUG [hconnection-0x3c456231-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,137 DEBUG [hconnection-0x451a10e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,137 DEBUG [hconnection-0x71e390a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,139 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,138 DEBUG [hconnection-0x1c1eef24-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,138 DEBUG [hconnection-0x695779ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,141 DEBUG [hconnection-0x4a68f21d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,141 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:29,142 DEBUG [hconnection-0x5053f2cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,142 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,142 DEBUG [hconnection-0x5a9b0863-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,142 DEBUG [hconnection-0x61ad9c20-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,143 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,146 DEBUG [hconnection-0x175186f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:29,152 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,152 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,153 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,153 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,153 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,153 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:29,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:29,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:29,157 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:29,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:29,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:29,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,158 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,159 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,159 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:29,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:29,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:29,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:29,161 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:29,161 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,161 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:29,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:29,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:58:29,165 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:29,165 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:29,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:29,166 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:29,166 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:29,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:29,166 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:29,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:29,166 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:29,166 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:29,166 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:29,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:29,166 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,166 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:29,166 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:29,173 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,173 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,178 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:29,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c6fa11[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:29,180 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:29,180 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803969179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803969179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,181 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803969180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,181 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:29,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803969180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,182 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:29,182 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:29,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,182 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:29,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803969181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,183 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,183 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:29,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:29,186 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:29,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:29,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:29,186 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:29,186 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:29,187 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,187 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,188 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,189 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,190 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,190 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,190 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:29,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:29,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,191 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:29,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,191 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:29,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803969282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803969282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803969282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803969282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803969283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,291 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:29,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:29,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:29,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:29,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:29,298 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:29,298 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,298 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:29,298 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:29,299 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,299 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,299 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,299 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,300 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:29,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:29,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:29,300 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:29,300 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,300 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:29,302 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:29,302 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:29,303 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,303 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:29,303 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:29,303 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,303 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:29,303 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:29,303 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,303 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:29,303 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,304 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7fbf1a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:29,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:29,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,306 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:29,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:29,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:29,307 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:29,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:29,309 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:29,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:29,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:29,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:29,310 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:29,311 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,311 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,311 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,312 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:29,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:29,321 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,321 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,321 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:29,322 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:29,422 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:29,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:29,425 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:29,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:29,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:29,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:29,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:29,427 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,427 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:29,427 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:29,428 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:29,428 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c140c0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:29,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:29,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:29,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:29,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:29,434 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:29,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:29,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:29,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:29,434 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:29,435 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,435 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,435 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,435 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,436 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:29,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,439 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,439 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,439 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,439 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,439 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,439 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:29,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,439 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:29,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:29,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803969484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803969484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803969485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803969485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803969486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,539 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:29,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:29,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:29,542 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:29,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:29,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:29,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,543 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:29,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:29,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:29,544 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,544 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:29,544 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:29,544 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:29,544 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:29,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,545 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:29,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:29,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6691b477[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:29,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:29,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,547 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:29,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:29,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:29,547 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,547 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:29,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:29,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:29,550 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:29,550 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:29,550 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:29,550 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:29,550 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:29,551 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,551 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,551 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,555 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:29,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:29,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,555 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,555 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:29,555 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:29,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/ad504c1fb8004530955d651136f15bd6 2023-04-06 17:58:29,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/c33ca04fc4db42359cb4b2bf8f747db1 2023-04-06 17:58:29,655 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:29,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:29,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:29,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:29,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:29,658 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:29,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:29,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:29,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:29,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:29,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:29,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:29,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:29,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:29,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:29,660 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,661 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,661 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:29,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3554dcc6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:29,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:29,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,662 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:29,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:29,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:29,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:29,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:29,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:29,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:29,665 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:29,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:29,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:29,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:29,665 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:29,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,666 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,666 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,668 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:29,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:29,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,669 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:29,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,669 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:29,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,769 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:29,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:29,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:29,772 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:29,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:29,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:29,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:29,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:29,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:29,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:29,775 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,776 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,777 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,777 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:29,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:29,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e02219[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:29,777 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:29,777 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,777 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,777 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:29,777 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:29,777 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:29,777 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,777 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:29,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:29,778 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:29,780 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:29,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:29,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:29,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:29,780 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:29,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,781 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,785 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:29,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,785 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,785 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:29,785 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803969789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803969790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803969790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803969790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803969791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,885 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:29,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:29,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:29,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:29,888 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:29,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:29,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:29,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,889 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,889 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,889 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:29,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:29,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:29,890 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,890 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:29,891 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:29,891 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:29,892 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:29,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6544b545[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:29,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:29,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,894 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:29,894 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:29,894 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:29,894 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:29,894 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:29,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:29,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:29,896 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:29,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:29,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:29,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:29,897 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:29,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:29,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:29,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,899 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:29,901 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:29,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:29,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:29,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:29,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:29,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:29,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,902 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:29,902 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:29,902 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:29,902 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:29,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/b22f1ff24d2440a7b4c0ff34c9ca1318 2023-04-06 17:58:29,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/ad504c1fb8004530955d651136f15bd6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad504c1fb8004530955d651136f15bd6 2023-04-06 17:58:29,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad504c1fb8004530955d651136f15bd6, entries=200, sequenceid=15, filesize=13.9 K 2023-04-06 17:58:29,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/c33ca04fc4db42359cb4b2bf8f747db1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c33ca04fc4db42359cb4b2bf8f747db1 2023-04-06 17:58:29,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c33ca04fc4db42359cb4b2bf8f747db1, entries=100, sequenceid=15, filesize=9.3 K 2023-04-06 17:58:29,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/b22f1ff24d2440a7b4c0ff34c9ca1318 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b22f1ff24d2440a7b4c0ff34c9ca1318 2023-04-06 17:58:30,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b22f1ff24d2440a7b4c0ff34c9ca1318, entries=100, sequenceid=15, filesize=9.3 K 2023-04-06 17:58:30,001 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for ec62d27e82d7188e5d149482edb0608d in 837ms, sequenceid=15, compaction requested=false 2023-04-06 17:58:30,001 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:30,002 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:30,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:30,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:30,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:30,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:30,004 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:30,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:30,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:30,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:30,005 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,005 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,005 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,005 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:30,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:30,005 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,006 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:30,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:30,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:30,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:30,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:30,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:30,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:30,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:30,008 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:30,008 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:30,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:30,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:30,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803970302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803970302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803970303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803970303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803970303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803970404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803970404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803970407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803970409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803970409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,417 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/75a242fac00f423f8e614834a9819b80 2023-04-06 17:58:30,425 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/0ee406bb9ae44bcc971ec464cbf30da4 2023-04-06 17:58:30,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803970608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803970608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803970610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803970612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803970614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,834 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/4de01fd894ce49ce8294306bcf337bd8 2023-04-06 17:58:30,837 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/75a242fac00f423f8e614834a9819b80 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/75a242fac00f423f8e614834a9819b80 2023-04-06 17:58:30,840 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/75a242fac00f423f8e614834a9819b80, entries=150, sequenceid=34, filesize=11.6 K 2023-04-06 17:58:30,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/0ee406bb9ae44bcc971ec464cbf30da4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/0ee406bb9ae44bcc971ec464cbf30da4 2023-04-06 17:58:30,843 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/0ee406bb9ae44bcc971ec464cbf30da4, entries=150, sequenceid=34, filesize=11.6 K 2023-04-06 17:58:30,844 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/4de01fd894ce49ce8294306bcf337bd8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/4de01fd894ce49ce8294306bcf337bd8 2023-04-06 17:58:30,846 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/4de01fd894ce49ce8294306bcf337bd8, entries=150, sequenceid=34, filesize=11.6 K 2023-04-06 17:58:30,847 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ec62d27e82d7188e5d149482edb0608d in 839ms, sequenceid=34, compaction requested=false 2023-04-06 17:58:30,847 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:30,847 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:30,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:30,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:30,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:30,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:30,848 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:30,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:30,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:30,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e9d9796[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:30,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:30,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:30,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:30,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:30,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:30,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:30,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:30,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:30,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:30,852 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:30,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:30,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:30,852 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:30,852 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:30,853 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:30,853 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,853 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:30,853 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:30,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:30,856 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:30,856 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:30,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:30,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 17:58:30,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:30,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:30,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:30,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:30,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:30,914 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:30,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/48d30531b7a04004ba06fd3845b27992 2023-04-06 17:58:30,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803970933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803970934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803970935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803970936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:30,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803970936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,956 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:30,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:30,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:30,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:30,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:30,959 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:30,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:30,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:30,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:30,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:30,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:30,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:30,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:30,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:30,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:30,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:30,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:30,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:30,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:30,962 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:30,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:30,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:30,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3cc86f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:30,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:30,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,964 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:30,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:30,964 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:30,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:30,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:30,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:30,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:30,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:30,966 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:30,966 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:30,966 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:30,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:30,967 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:30,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:30,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,969 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:30,970 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:30,971 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:30,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:30,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:30,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:30,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:30,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:30,971 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:30,971 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:31,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803971037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803971041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803971041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803971041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803971042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,072 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:31,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:31,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:31,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:31,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:31,074 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:31,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:31,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:31,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,074 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,075 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:31,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:31,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:31,076 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:31,076 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:31,076 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:31,076 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:31,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:31,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:31,078 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:31,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a4bd13b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:31,078 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,079 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,079 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:31,079 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:31,079 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:31,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:31,080 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,080 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,080 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,080 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:31,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:31,081 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:31,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:31,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:31,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:31,081 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:31,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,082 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,085 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,085 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:31,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:31,085 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,086 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:31,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,086 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:31,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:31,189 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:31,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:31,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:31,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:31,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:31,191 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:31,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:31,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:31,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,192 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:31,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:31,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:31,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:31,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:31,194 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,194 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:31,194 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:31,194 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:31,195 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,195 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,195 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,195 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:31,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:31,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a65b1aa[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:31,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,197 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:31,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:31,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:31,197 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,197 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:31,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:31,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,198 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:31,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:31,200 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:31,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:31,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:31,200 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:31,200 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:31,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,201 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,202 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:31,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:31,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:31,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,204 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,204 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:31,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,204 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:31,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803971240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803971248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803971249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803971249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803971249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,304 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:31,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:31,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:31,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:31,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:31,306 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,307 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:31,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:31,308 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:31,308 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,308 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:31,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:31,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:31,309 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,309 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:31,309 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:31,309 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:31,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:31,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:31,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:31,310 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,314 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:31,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38a6bc57[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:31,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:31,317 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:31,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:31,323 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,323 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:31,323 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:31,324 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,324 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:31,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:31,326 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:31,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:31,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:31,326 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:31,326 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:31,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,327 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,327 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,328 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,330 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:31,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:31,330 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:31,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:31,330 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,331 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/c7e544ea823a4025a7d9f35fa8e5fcf9 2023-04-06 17:58:31,430 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:31,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:31,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:31,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:31,433 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:31,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:31,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:31,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,434 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:31,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:31,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:31,435 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,435 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:31,435 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:31,435 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:31,436 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,436 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,437 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,437 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:31,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:31,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4322649c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:31,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:31,439 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:31,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:31,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:31,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:31,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,440 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:31,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:31,443 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:31,444 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:31,444 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:31,444 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:31,444 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:31,444 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,444 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,444 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,445 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,447 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,449 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:31,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,449 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,449 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,449 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,549 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:31,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:31,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803971546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:31,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:31,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:31,552 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:31,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:31,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:31,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,552 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:31,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:31,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:31,554 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,554 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:31,554 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:31,554 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:31,554 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:31,555 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,555 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,555 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:31,556 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:31,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724a7ca[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:31,557 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:31,557 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,557 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,557 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,558 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:31,557 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:31,558 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:31,558 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:31,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803971554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803971554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803971555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,558 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:31,558 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:31,558 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803971555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,558 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:31,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:31,560 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:31,560 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:31,560 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:31,560 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:31,560 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:31,561 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,561 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,561 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,561 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,563 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,564 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:31,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,564 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:31,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,564 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,564 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,664 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:31,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:31,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:31,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:31,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:31,667 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:31,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:31,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:31,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,668 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,668 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,668 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:31,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:31,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:31,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:31,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:31,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:31,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:31,669 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,669 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:31,669 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:31,669 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:31,669 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:31,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:31,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:31,670 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,670 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,671 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:31,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bbbe65[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:31,672 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:31,672 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,672 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,672 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,672 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:31,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:31,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:31,672 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:31,673 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,673 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:31,673 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,673 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:31,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:31,677 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:31,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:31,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:31,677 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:31,677 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:31,677 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,677 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:31,677 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:31,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,678 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,680 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,680 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,680 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,680 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,680 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:31,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:31,680 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:31,680 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:31,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:31,680 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:31,680 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:31,680 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,681 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:31,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:31,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/620710eff8c84975b469e826d090eade 2023-04-06 17:58:31,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/48d30531b7a04004ba06fd3845b27992 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d30531b7a04004ba06fd3845b27992 2023-04-06 17:58:31,752 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d30531b7a04004ba06fd3845b27992, entries=200, sequenceid=48, filesize=13.9 K 2023-04-06 17:58:31,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/c7e544ea823a4025a7d9f35fa8e5fcf9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c7e544ea823a4025a7d9f35fa8e5fcf9 2023-04-06 17:58:31,755 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c7e544ea823a4025a7d9f35fa8e5fcf9, entries=150, sequenceid=48, filesize=11.6 K 2023-04-06 17:58:31,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/620710eff8c84975b469e826d090eade as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/620710eff8c84975b469e826d090eade 2023-04-06 17:58:31,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/620710eff8c84975b469e826d090eade, entries=150, sequenceid=48, filesize=11.6 K 2023-04-06 17:58:31,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ec62d27e82d7188e5d149482edb0608d in 845ms, sequenceid=48, compaction requested=true 2023-04-06 17:58:31,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:31,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:31,759 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:31,759 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:31,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:31,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:31,760 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40334 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:31,760 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:31,760 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:31,760 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:31,760 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,760 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,760 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad504c1fb8004530955d651136f15bd6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/75a242fac00f423f8e614834a9819b80, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d30531b7a04004ba06fd3845b27992] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=39.4 K 2023-04-06 17:58:31,760 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c33ca04fc4db42359cb4b2bf8f747db1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/0ee406bb9ae44bcc971ec464cbf30da4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c7e544ea823a4025a7d9f35fa8e5fcf9] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=32.5 K 2023-04-06 17:58:31,761 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ad504c1fb8004530955d651136f15bd6, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680803909160 2023-04-06 17:58:31,761 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c33ca04fc4db42359cb4b2bf8f747db1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680803909164 2023-04-06 17:58:31,761 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 75a242fac00f423f8e614834a9819b80, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680803909176 2023-04-06 17:58:31,761 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0ee406bb9ae44bcc971ec464cbf30da4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680803909176 2023-04-06 17:58:31,761 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 48d30531b7a04004ba06fd3845b27992, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803910301 2023-04-06 17:58:31,761 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c7e544ea823a4025a7d9f35fa8e5fcf9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803910301 2023-04-06 17:58:31,776 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#407 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:31,776 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:31,780 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:31,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:31,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:31,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:31,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:31,790 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:31,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:31,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:31,791 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,791 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,791 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,791 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:31,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:31,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:31,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:31,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:31,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:31,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:31,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:31,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:31,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:31,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:31,794 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,794 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:31,794 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:31,794 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:58:31,794 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:31,794 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:31,795 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:31,795 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:31,795 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:31,795 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:31,799 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/3a7fe282016d4d3f91a741d9f475aaf2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a7fe282016d4d3f91a741d9f475aaf2 2023-04-06 17:58:31,803 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into 3a7fe282016d4d3f91a741d9f475aaf2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:31,803 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:31,803 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=13, startTime=1680803911759; duration=0sec 2023-04-06 17:58:31,803 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:31,803 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:31,804 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:31,804 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:31,804 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:31,804 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b22f1ff24d2440a7b4c0ff34c9ca1318, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/4de01fd894ce49ce8294306bcf337bd8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/620710eff8c84975b469e826d090eade] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=32.5 K 2023-04-06 17:58:31,804 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b22f1ff24d2440a7b4c0ff34c9ca1318, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680803909164 2023-04-06 17:58:31,804 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4de01fd894ce49ce8294306bcf337bd8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680803909176 2023-04-06 17:58:31,805 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 620710eff8c84975b469e826d090eade, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803910301 2023-04-06 17:58:31,806 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/48d183ed3c614b218eb13817af3e26d2 2023-04-06 17:58:31,809 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:31,826 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/a43572275b37467bb7fe3e1cb8726b7c 2023-04-06 17:58:32,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:32,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:32,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803972092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803972093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803972093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803972094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803972094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,112 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 17:58:32,195 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/af77859c16da45678c2d6de83c6c1e19 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/af77859c16da45678c2d6de83c6c1e19 2023-04-06 17:58:32,197 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into af77859c16da45678c2d6de83c6c1e19(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:32,198 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:32,198 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=13, startTime=1680803911759; duration=0sec 2023-04-06 17:58:32,198 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:32,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803972199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803972200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803972204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803972204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803972204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,218 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/6c45d7c799024603936b821e2c851ecc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6c45d7c799024603936b821e2c851ecc 2023-04-06 17:58:32,221 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into 6c45d7c799024603936b821e2c851ecc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:32,221 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:32,221 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=13, startTime=1680803911759; duration=0sec 2023-04-06 17:58:32,221 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:32,233 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/efed95fd3eed449896bc12456a52aaa5 2023-04-06 17:58:32,235 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/48d183ed3c614b218eb13817af3e26d2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d183ed3c614b218eb13817af3e26d2 2023-04-06 17:58:32,238 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d183ed3c614b218eb13817af3e26d2, entries=150, sequenceid=66, filesize=11.6 K 2023-04-06 17:58:32,238 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/a43572275b37467bb7fe3e1cb8726b7c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/a43572275b37467bb7fe3e1cb8726b7c 2023-04-06 17:58:32,241 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/a43572275b37467bb7fe3e1cb8726b7c, entries=150, sequenceid=66, filesize=11.6 K 2023-04-06 17:58:32,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/efed95fd3eed449896bc12456a52aaa5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/efed95fd3eed449896bc12456a52aaa5 2023-04-06 17:58:32,245 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/efed95fd3eed449896bc12456a52aaa5, entries=150, sequenceid=66, filesize=11.6 K 2023-04-06 17:58:32,245 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for ec62d27e82d7188e5d149482edb0608d in 451ms, sequenceid=66, compaction requested=false 2023-04-06 17:58:32,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:32,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:32,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:32,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:32,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:32,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:32,248 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:32,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d72e021[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:32,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:32,250 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,250 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:32,251 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:32,251 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:32,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,251 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:32,253 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:32,253 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:32,253 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:32,253 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:32,253 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:32,253 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,253 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:32,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,254 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:32,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:32,254 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:32,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:32,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:32,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:32,265 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,265 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:32,265 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:32,265 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,265 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,265 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:32,265 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:32,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:32,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:32,265 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:32,265 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:32,366 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:32,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:32,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:32,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:32,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:32,368 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:32,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:32,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:32,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:32,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,368 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:32,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,369 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:32,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:32,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:32,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:32,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:32,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:32,370 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:32,370 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:32,370 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 17:58:32,370 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:32,371 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:32,371 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:32,371 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:32,371 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:32,371 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:32,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:32,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:32,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803972423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803972424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803972424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803972425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803972426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803972527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803972531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803972531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803972533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803972534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803972729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803972735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803972736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803972739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:32,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803972739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:32,778 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/41cc90541de8454b9fb9235f3caab84c 2023-04-06 17:58:33,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803973035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803973039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803973039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803973044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803973044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,185 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/815046ac35c045de89f0c4a3e99edcfc 2023-04-06 17:58:33,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803973541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803973545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803973548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803973550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:33,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803973551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,595 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/8b0dcae0f7474c1ca7af6fc35de33d2b 2023-04-06 17:58:33,598 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/41cc90541de8454b9fb9235f3caab84c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/41cc90541de8454b9fb9235f3caab84c 2023-04-06 17:58:33,601 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/41cc90541de8454b9fb9235f3caab84c, entries=150, sequenceid=84, filesize=11.6 K 2023-04-06 17:58:33,601 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/815046ac35c045de89f0c4a3e99edcfc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/815046ac35c045de89f0c4a3e99edcfc 2023-04-06 17:58:33,604 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/815046ac35c045de89f0c4a3e99edcfc, entries=150, sequenceid=84, filesize=11.6 K 2023-04-06 17:58:33,605 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/8b0dcae0f7474c1ca7af6fc35de33d2b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/8b0dcae0f7474c1ca7af6fc35de33d2b 2023-04-06 17:58:33,607 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/8b0dcae0f7474c1ca7af6fc35de33d2b, entries=150, sequenceid=84, filesize=11.6 K 2023-04-06 17:58:33,608 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for ec62d27e82d7188e5d149482edb0608d in 1238ms, sequenceid=84, compaction requested=true 2023-04-06 17:58:33,609 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:33,609 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:33,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:33,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:33,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:33,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:33,610 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,611 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,611 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:33,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d246d97[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:33,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:33,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,612 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:33,612 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:33,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:33,612 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:33,613 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:33,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,613 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,613 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:33,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:33,615 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:33,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:33,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:33,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:33,615 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:33,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:33,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:33,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,620 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:33,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:33,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:33,620 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,620 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:33,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:33,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,720 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:33,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:33,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:33,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:33,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:33,722 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:33,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:33,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:33,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:33,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:33,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:33,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:33,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:33,725 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:33,725 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:33,725 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:58:33,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:33,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:33,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:33,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:33,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:33,726 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:33,734 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/7abc6ccf15f0465bb05bfac8166aefb8 2023-04-06 17:58:33,741 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/474ccbfe69414315a279cb52750e9946 2023-04-06 17:58:33,756 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/ad97722813d04c7994f9f9f247c341d1 2023-04-06 17:58:33,760 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/7abc6ccf15f0465bb05bfac8166aefb8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/7abc6ccf15f0465bb05bfac8166aefb8 2023-04-06 17:58:33,764 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/7abc6ccf15f0465bb05bfac8166aefb8, entries=150, sequenceid=102, filesize=11.6 K 2023-04-06 17:58:33,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/474ccbfe69414315a279cb52750e9946 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/474ccbfe69414315a279cb52750e9946 2023-04-06 17:58:33,767 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/474ccbfe69414315a279cb52750e9946, entries=150, sequenceid=102, filesize=11.6 K 2023-04-06 17:58:33,767 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/ad97722813d04c7994f9f9f247c341d1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ad97722813d04c7994f9f9f247c341d1 2023-04-06 17:58:33,770 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ad97722813d04c7994f9f9f247c341d1, entries=150, sequenceid=102, filesize=11.6 K 2023-04-06 17:58:33,771 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for ec62d27e82d7188e5d149482edb0608d in 46ms, sequenceid=102, compaction requested=true 2023-04-06 17:58:33,771 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:33,771 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:33,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:33,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:33,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:33,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:33,772 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,772 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,772 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,772 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:33,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:33,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f9c850e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:33,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:33,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:33,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:33,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:33,775 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,775 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,775 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,775 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:33,779 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:33,779 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:33,779 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:33,779 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:33,779 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:33,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,780 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:33,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,781 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,782 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:33,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:33,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:33,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:33,783 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:33,783 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:33,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:33,883 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:33,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:33,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:33,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:33,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:33,886 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:33,886 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:33,886 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:33,886 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:33,886 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,886 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,887 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:33,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:33,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:33,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:33,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:33,888 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:33,888 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:33,888 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:33,888 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:33,888 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:33,888 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:33,888 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:33,889 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:33,889 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:33,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:33,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:33,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:33,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:33,890 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:33,891 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f94a0e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:33,891 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:33,892 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,892 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:33,893 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,893 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:33,893 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,893 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:33,895 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:33,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:33,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:33,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:33,895 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:33,897 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:33,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:33,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:33,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:33,898 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:33,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:33,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:33,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:33,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:33,900 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:33,900 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:33,900 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,000 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,003 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4860c598[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,010 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,010 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,010 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,014 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,014 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,017 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,018 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:34,020 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,020 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,120 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,124 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,125 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,128 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,128 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,128 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:34,128 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,129 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,130 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@462651d6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,131 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,131 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,132 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,132 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,132 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,135 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,135 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,135 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,135 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,135 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,136 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,137 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,137 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,139 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,140 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,140 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,140 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,240 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,243 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,246 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,246 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,246 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:34,246 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,247 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,247 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,247 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,247 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2df918e9[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,250 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,250 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,252 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,252 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,252 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,252 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,252 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,253 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,254 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,256 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,256 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,256 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:34,356 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,359 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,359 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,360 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,361 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,361 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,361 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:34,361 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,361 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,362 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,362 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,364 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45388e62[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,364 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,364 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,365 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,365 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,365 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,365 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,366 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,367 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,367 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,367 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,369 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,369 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,369 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,369 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,370 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,372 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,372 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:34,472 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,475 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,475 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,476 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,476 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,476 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,476 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,477 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,477 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,477 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:34,477 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,477 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,478 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,478 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,478 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,478 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b3acf58[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,479 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,479 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,480 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,481 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,481 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,481 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,482 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,482 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,484 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,484 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,484 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,485 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,487 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,489 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,489 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:34,564 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:58:34,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:34,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:34,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:34,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:34,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:34,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:34,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/8eb5630bf03b4e618dcc00646c536242 2023-04-06 17:58:34,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/b599f7187f1b47fe86b84918e47d087c 2023-04-06 17:58:34,589 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,592 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,592 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,593 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,593 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,594 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,594 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,594 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:34,594 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,594 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,595 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe68708[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,596 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,596 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,598 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,599 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,600 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,600 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,601 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,601 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,601 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803974589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803974589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803974597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,604 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,604 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803974597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803974602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,704 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803974702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803974703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803974703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,707 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,707 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,708 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:34,709 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,709 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,710 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,711 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3de64d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,711 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,712 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,712 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,712 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803974708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803974709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,713 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,720 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,720 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,732 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,732 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,832 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,836 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,837 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:34,838 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf720f6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,840 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,840 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,843 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,843 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,847 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:34,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,847 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,847 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803974906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803974907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803974907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803974914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803974914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,947 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:34,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:34,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:34,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:34,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:34,950 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:34,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:34,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:34,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,951 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,951 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:34,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:34,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:34,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:34,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:34,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:34,952 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:34,952 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:34,953 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:34,954 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,954 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:34,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466a6cdf[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:34,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:34,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:34,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:34,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:34,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:34,957 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:34,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,957 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:34,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:34,959 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:34,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:34,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:34,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:34,959 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:34,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:34,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:34,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:34,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,962 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,962 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:34,962 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,962 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:34,962 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:34,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:34,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:34,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:34,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:34,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:34,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:34,962 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:34,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:34,963 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:34,963 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:34,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/01469994e3a7411e8587ff721fb6e668 2023-04-06 17:58:34,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/8eb5630bf03b4e618dcc00646c536242 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/8eb5630bf03b4e618dcc00646c536242 2023-04-06 17:58:34,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/8eb5630bf03b4e618dcc00646c536242, entries=250, sequenceid=120, filesize=16.2 K 2023-04-06 17:58:34,996 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/b599f7187f1b47fe86b84918e47d087c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b599f7187f1b47fe86b84918e47d087c 2023-04-06 17:58:34,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b599f7187f1b47fe86b84918e47d087c, entries=150, sequenceid=120, filesize=11.6 K 2023-04-06 17:58:34,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/01469994e3a7411e8587ff721fb6e668 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/01469994e3a7411e8587ff721fb6e668 2023-04-06 17:58:35,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/01469994e3a7411e8587ff721fb6e668, entries=150, sequenceid=120, filesize=11.6 K 2023-04-06 17:58:35,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ec62d27e82d7188e5d149482edb0608d in 438ms, sequenceid=120, compaction requested=true 2023-04-06 17:58:35,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:35,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:35,003 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:58:35,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:35,003 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:58:35,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:35,004 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64213 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:58:35,004 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59533 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:58:35,004 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:35,004 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:35,004 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,004 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,004 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/af77859c16da45678c2d6de83c6c1e19, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d183ed3c614b218eb13817af3e26d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/41cc90541de8454b9fb9235f3caab84c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/7abc6ccf15f0465bb05bfac8166aefb8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/8eb5630bf03b4e618dcc00646c536242] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=62.7 K 2023-04-06 17:58:35,004 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a7fe282016d4d3f91a741d9f475aaf2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/a43572275b37467bb7fe3e1cb8726b7c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/815046ac35c045de89f0c4a3e99edcfc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/474ccbfe69414315a279cb52750e9946, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b599f7187f1b47fe86b84918e47d087c] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=58.1 K 2023-04-06 17:58:35,004 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting af77859c16da45678c2d6de83c6c1e19, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803910301 2023-04-06 17:58:35,004 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3a7fe282016d4d3f91a741d9f475aaf2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803910301 2023-04-06 17:58:35,004 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 48d183ed3c614b218eb13817af3e26d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680803910932 2023-04-06 17:58:35,005 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a43572275b37467bb7fe3e1cb8726b7c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680803910932 2023-04-06 17:58:35,005 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 41cc90541de8454b9fb9235f3caab84c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680803912092 2023-04-06 17:58:35,005 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 815046ac35c045de89f0c4a3e99edcfc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680803912092 2023-04-06 17:58:35,005 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 474ccbfe69414315a279cb52750e9946, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680803912422 2023-04-06 17:58:35,005 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 7abc6ccf15f0465bb05bfac8166aefb8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680803912422 2023-04-06 17:58:35,005 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b599f7187f1b47fe86b84918e47d087c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680803914561 2023-04-06 17:58:35,006 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 8eb5630bf03b4e618dcc00646c536242, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680803914559 2023-04-06 17:58:35,012 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:35,014 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#422 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:35,020 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/aae5893adf974d6592027994d596f536 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/aae5893adf974d6592027994d596f536 2023-04-06 17:58:35,025 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into aae5893adf974d6592027994d596f536(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:35,025 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:35,025 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=11, startTime=1680803915003; duration=0sec 2023-04-06 17:58:35,025 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:35,025 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 17:58:35,026 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59533 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 17:58:35,027 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:35,027 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,027 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6c45d7c799024603936b821e2c851ecc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/efed95fd3eed449896bc12456a52aaa5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/8b0dcae0f7474c1ca7af6fc35de33d2b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ad97722813d04c7994f9f9f247c341d1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/01469994e3a7411e8587ff721fb6e668] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=58.1 K 2023-04-06 17:58:35,027 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6c45d7c799024603936b821e2c851ecc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680803910301 2023-04-06 17:58:35,027 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting efed95fd3eed449896bc12456a52aaa5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680803910932 2023-04-06 17:58:35,028 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 8b0dcae0f7474c1ca7af6fc35de33d2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680803912092 2023-04-06 17:58:35,028 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ad97722813d04c7994f9f9f247c341d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680803912422 2023-04-06 17:58:35,028 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 01469994e3a7411e8587ff721fb6e668, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680803914561 2023-04-06 17:58:35,036 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#423 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:35,038 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/cd2eefa1650d4500a598a7641b28f496 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/cd2eefa1650d4500a598a7641b28f496 2023-04-06 17:58:35,042 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into cd2eefa1650d4500a598a7641b28f496(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:35,042 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:35,042 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=11, startTime=1680803915003; duration=0sec 2023-04-06 17:58:35,042 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:35,063 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:35,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:35,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:35,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:35,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:35,066 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:35,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:35,066 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:35,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,067 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,068 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,068 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:35,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:35,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:35,074 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:35,074 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:35,074 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,074 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:35,074 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:58:35,074 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:35,075 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:35,075 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:35,075 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:35,075 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:35,075 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:35,083 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e 2023-04-06 17:58:35,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:35,214 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:35,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803975223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803975227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803975227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803975228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803975230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803975334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803975334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803975334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803975334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,458 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/ac4c3108855748b4a9c5b7fd5aa2c052 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ac4c3108855748b4a9c5b7fd5aa2c052 2023-04-06 17:58:35,461 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into ac4c3108855748b4a9c5b7fd5aa2c052(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:35,461 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:35,461 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=11, startTime=1680803915003; duration=0sec 2023-04-06 17:58:35,462 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:35,500 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/2e4f3ef60a8f4617a70691ad1fb6b3c5 2023-04-06 17:58:35,507 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/69cb2560fa84419e8445be2aad3fc6c5 2023-04-06 17:58:35,511 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e 2023-04-06 17:58:35,513 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e, entries=150, sequenceid=142, filesize=11.8 K 2023-04-06 17:58:35,514 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/2e4f3ef60a8f4617a70691ad1fb6b3c5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/2e4f3ef60a8f4617a70691ad1fb6b3c5 2023-04-06 17:58:35,518 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/2e4f3ef60a8f4617a70691ad1fb6b3c5, entries=150, sequenceid=142, filesize=11.8 K 2023-04-06 17:58:35,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/69cb2560fa84419e8445be2aad3fc6c5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/69cb2560fa84419e8445be2aad3fc6c5 2023-04-06 17:58:35,522 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/69cb2560fa84419e8445be2aad3fc6c5, entries=150, sequenceid=142, filesize=11.8 K 2023-04-06 17:58:35,523 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ec62d27e82d7188e5d149482edb0608d in 448ms, sequenceid=142, compaction requested=false 2023-04-06 17:58:35,523 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:35,523 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:35,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:35,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dff6ddd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:35,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:35,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,526 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:35,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:35,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:35,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,526 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:35,526 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:35,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:35,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:35,529 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:35,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:35,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:35,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:35,529 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:35,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,530 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:35,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:35,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,533 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,533 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,533 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:35,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:35,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 17:58:35,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:35,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:35,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:35,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:35,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:35,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:35,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803975558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803975560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803975563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803975564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,634 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:35,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:35,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:35,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:35,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:35,636 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:35,636 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,636 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:35,636 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:35,636 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,637 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:35,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:35,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:35,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:35,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:35,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:35,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:35,639 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,639 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:35,639 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:35,639 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:35,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:35,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:35,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:35,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:35,642 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:35,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7885fda1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:35,642 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,642 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:35,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:35,643 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,643 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:35,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:35,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:35,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,643 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:35,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:35,645 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:35,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:35,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:35,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:35,645 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:35,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:35,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:35,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:35,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,649 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:35,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,649 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:35,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803975665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803975665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803975666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803975667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803975732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,749 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:35,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:35,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:35,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:35,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:35,752 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:35,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:35,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:35,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,752 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,753 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:35,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:35,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:35,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:35,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:35,754 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,754 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:35,754 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:35,754 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:35,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:35,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:35,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:35,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:35,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad782d7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:35,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:35,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,757 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,757 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:35,757 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:35,757 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:35,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:35,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:35,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:35,758 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:35,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:35,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:35,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:35,758 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:35,759 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,759 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,759 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:35,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:35,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,763 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:35,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:35,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,763 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:35,764 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,764 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,864 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:35,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:35,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:35,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:35,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:35,867 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:35,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:35,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:35,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,867 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:35,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:35,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:35,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:35,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:35,869 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:35,869 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,869 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:35,869 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:35,869 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,869 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:35,869 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:35,870 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:35,870 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:35,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:35,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:35,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:35,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,870 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:35,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4e11c6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:35,872 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:35,872 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:35,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:35,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:35,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:35,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:35,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:35,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803975869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803975869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803975869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:35,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803975872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:35,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:35,876 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:35,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:35,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:35,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:35,876 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:35,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,878 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,880 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:35,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,881 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,881 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:35,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:35,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/0ec976034a3e4066b2ac7257775c709b 2023-04-06 17:58:35,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/efae3397841e422fbaa92a56af72c820 2023-04-06 17:58:35,981 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:35,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:35,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:35,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:35,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:35,985 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:35,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:35,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:35,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,988 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,988 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,988 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:35,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:35,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:35,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:35,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:35,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:35,989 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:35,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:35,990 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,990 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:35,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a91c77[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:35,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:35,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,992 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:35,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:35,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:35,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:35,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:35,993 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:35,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:35,995 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:35,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:35,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:35,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:35,995 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:35,996 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:35,996 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,996 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:35,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:35,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:35,997 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:35,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:35,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:35,998 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:35,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:35,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:35,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:35,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:35,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:35,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:35,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:35,999 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:35,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:35,999 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:36,104 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:36,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:36,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:36,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:36,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:36,107 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:36,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:36,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:36,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,107 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:36,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:36,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:36,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:36,109 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,109 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:36,109 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:36,109 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:36,109 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:36,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:36,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:36,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e169616[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:36,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:36,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:36,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:36,112 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,112 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:36,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:36,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:36,115 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:36,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:36,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:36,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:36,115 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:36,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:36,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:36,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:36,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:36,121 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:36,121 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:36,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:36,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:36,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803976175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803976175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803976176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803976177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,221 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:36,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:36,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:36,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:36,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:36,223 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:36,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:36,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:36,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,224 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:36,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:36,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:36,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:36,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:36,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:36,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:36,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:36,225 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,225 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:36,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:36,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:36,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:36,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:36,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:36,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:36,226 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,227 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,227 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,227 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:36,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:36,228 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:36,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8316039[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:36,228 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,228 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:36,229 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,229 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:36,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:36,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:36,229 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:36,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:36,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:36,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:36,231 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:36,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:36,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:36,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:36,231 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:36,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:36,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,233 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:36,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:36,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:36,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:36,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:36,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,236 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,236 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:36,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,236 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,336 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:36,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:36,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:36,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:36,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:36,339 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:36,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:36,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:36,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,340 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:36,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:36,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:36,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:36,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:36,341 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:36,341 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:36,341 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:36,341 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:36,341 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:36,342 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:36,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:36,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374e90d8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:36,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,344 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:36,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:36,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:36,344 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,344 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:36,345 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:36,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:36,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:36,347 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:36,347 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:36,347 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:36,347 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:36,347 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:36,347 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,347 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:36,347 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:36,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:36,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:36,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:36,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:36,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:36,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,351 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:36,351 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:36,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:36,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/44bf4eec7927414aad42ddce646f79e8 2023-04-06 17:58:36,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/0ec976034a3e4066b2ac7257775c709b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0ec976034a3e4066b2ac7257775c709b 2023-04-06 17:58:36,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0ec976034a3e4066b2ac7257775c709b, entries=200, sequenceid=159, filesize=14.1 K 2023-04-06 17:58:36,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/efae3397841e422fbaa92a56af72c820 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/efae3397841e422fbaa92a56af72c820 2023-04-06 17:58:36,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/efae3397841e422fbaa92a56af72c820, entries=150, sequenceid=159, filesize=11.8 K 2023-04-06 17:58:36,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/44bf4eec7927414aad42ddce646f79e8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/44bf4eec7927414aad42ddce646f79e8 2023-04-06 17:58:36,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/44bf4eec7927414aad42ddce646f79e8, entries=150, sequenceid=159, filesize=11.8 K 2023-04-06 17:58:36,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ec62d27e82d7188e5d149482edb0608d in 846ms, sequenceid=159, compaction requested=true 2023-04-06 17:58:36,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:36,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:36,387 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:36,387 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:36,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:36,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:36,388 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38622 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:36,388 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:36,388 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,388 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/aae5893adf974d6592027994d596f536, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0ec976034a3e4066b2ac7257775c709b] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=37.7 K 2023-04-06 17:58:36,388 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36232 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:36,388 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:36,388 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,388 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting aae5893adf974d6592027994d596f536, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680803914561 2023-04-06 17:58:36,388 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/cd2eefa1650d4500a598a7641b28f496, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/2e4f3ef60a8f4617a70691ad1fb6b3c5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/efae3397841e422fbaa92a56af72c820] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=35.4 K 2023-04-06 17:58:36,388 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 57cc4ec5a1cb4e05a0c08dde6d5d3d7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680803914586 2023-04-06 17:58:36,388 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting cd2eefa1650d4500a598a7641b28f496, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680803914561 2023-04-06 17:58:36,389 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 0ec976034a3e4066b2ac7257775c709b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680803915213 2023-04-06 17:58:36,389 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2e4f3ef60a8f4617a70691ad1fb6b3c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680803914586 2023-04-06 17:58:36,389 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting efae3397841e422fbaa92a56af72c820, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680803915229 2023-04-06 17:58:36,394 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#430 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:36,394 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:36,401 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/f0d4881c18dd4ad395b43f423e70294c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/f0d4881c18dd4ad395b43f423e70294c 2023-04-06 17:58:36,406 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into f0d4881c18dd4ad395b43f423e70294c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:36,406 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:36,406 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=13, startTime=1680803916387; duration=0sec 2023-04-06 17:58:36,406 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:36,406 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:36,407 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36232 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:36,407 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:36,407 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,407 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ac4c3108855748b4a9c5b7fd5aa2c052, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/69cb2560fa84419e8445be2aad3fc6c5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/44bf4eec7927414aad42ddce646f79e8] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=35.4 K 2023-04-06 17:58:36,408 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ac4c3108855748b4a9c5b7fd5aa2c052, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680803914561 2023-04-06 17:58:36,408 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 69cb2560fa84419e8445be2aad3fc6c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680803914586 2023-04-06 17:58:36,408 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 44bf4eec7927414aad42ddce646f79e8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680803915229 2023-04-06 17:58:36,414 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#432 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:36,451 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:36,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:36,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:36,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:36,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:36,453 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:36,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:36,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:36,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,453 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,454 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:36,454 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:36,454 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,454 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:36,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:36,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:36,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:36,455 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:36,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:36,456 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 17:58:36,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:36,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:36,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:36,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:36,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:36,456 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:36,465 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/e64820b95f5f4c2f8bc7762788f49629 2023-04-06 17:58:36,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:36,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:36,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803976698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803976699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803976704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803976705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803976740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,801 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/4abbfd26fb21430fb45f235c423b0a90 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4abbfd26fb21430fb45f235c423b0a90 2023-04-06 17:58:36,804 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into 4abbfd26fb21430fb45f235c423b0a90(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:36,804 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:36,804 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=13, startTime=1680803916387; duration=0sec 2023-04-06 17:58:36,804 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:36,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803976808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803976808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803976810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:36,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803976810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:36,824 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/3e268ac4e0db45218dc92a6e63e418aa as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e268ac4e0db45218dc92a6e63e418aa 2023-04-06 17:58:36,828 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into 3e268ac4e0db45218dc92a6e63e418aa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:36,828 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:36,829 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=13, startTime=1680803916387; duration=0sec 2023-04-06 17:58:36,829 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:36,879 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/b467a484ba9f4790bd30b01436d83949 2023-04-06 17:58:37,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803977012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803977013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803977014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803977014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,288 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/e7c9a7d83d0b4f5b837abb6cd16d8929 2023-04-06 17:58:37,291 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/e64820b95f5f4c2f8bc7762788f49629 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e64820b95f5f4c2f8bc7762788f49629 2023-04-06 17:58:37,294 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e64820b95f5f4c2f8bc7762788f49629, entries=150, sequenceid=176, filesize=11.8 K 2023-04-06 17:58:37,295 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/b467a484ba9f4790bd30b01436d83949 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b467a484ba9f4790bd30b01436d83949 2023-04-06 17:58:37,297 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b467a484ba9f4790bd30b01436d83949, entries=150, sequenceid=176, filesize=11.8 K 2023-04-06 17:58:37,298 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/e7c9a7d83d0b4f5b837abb6cd16d8929 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e7c9a7d83d0b4f5b837abb6cd16d8929 2023-04-06 17:58:37,300 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e7c9a7d83d0b4f5b837abb6cd16d8929, entries=150, sequenceid=176, filesize=11.8 K 2023-04-06 17:58:37,301 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ec62d27e82d7188e5d149482edb0608d in 845ms, sequenceid=176, compaction requested=false 2023-04-06 17:58:37,301 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:37,301 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,301 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:37,301 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:37,301 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,301 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:37,301 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:37,302 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,302 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,302 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,302 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:37,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:37,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46f578e2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:37,303 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,304 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:37,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:37,304 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:37,304 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,304 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:37,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,305 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,305 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,305 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:37,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:37,306 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:37,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:37,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:37,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:37,307 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:37,307 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,307 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,307 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:37,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,310 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,310 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:37,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:37,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:37,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 17:58:37,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:37,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:37,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:37,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:37,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:37,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:37,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803977330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803977354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803977354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803977354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,410 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:37,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:37,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:37,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:37,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:37,413 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:37,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:37,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:37,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,414 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,414 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,414 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:37,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:37,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:37,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:37,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:37,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:37,415 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:37,415 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:37,416 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:37,416 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:37,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:37,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:37,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:37,417 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:37,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c89e00c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:37,418 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:37,418 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,419 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:37,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:37,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:37,419 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,419 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:37,419 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:37,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:37,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:37,421 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:37,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:37,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:37,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:37,421 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:37,422 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,422 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,422 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,422 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:37,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:37,426 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:37,426 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:37,426 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:37,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,427 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,427 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,427 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803977455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803977459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803977459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803977459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,526 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:37,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:37,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:37,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:37,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:37,529 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:37,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:37,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:37,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,530 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:37,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:37,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:37,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:37,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:37,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:37,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:37,532 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,532 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:37,532 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:37,532 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:37,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:37,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:37,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:37,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fba7b0c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:37,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:37,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:37,535 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:37,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:37,536 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:37,536 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,536 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,536 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:37,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:37,538 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:37,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:37,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:37,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:37,538 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:37,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,539 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,539 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,540 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:37,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,542 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,541 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:37,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:37,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,542 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:37,542 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:37,642 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:37,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:37,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:37,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:37,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:37,645 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:37,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:37,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:37,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:37,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:37,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:37,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:37,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:37,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:37,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:37,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:37,647 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,647 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:37,647 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:37,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:37,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:37,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:37,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:37,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:37,648 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,649 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,649 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:37,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:37,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:37,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c31e073[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:37,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,650 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:37,650 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:37,650 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:37,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:37,653 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:37,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:37,655 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:37,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:37,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:37,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:37,655 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:37,656 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,656 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,656 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:37,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:37,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:37,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,659 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:37,659 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:37,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803977657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803977661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803977662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803977662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/81bfa87e01214cd8bf62399916fbee84 2023-04-06 17:58:37,759 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:37,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:37,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:37,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:37,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:37,762 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:37,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:37,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:37,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,763 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:37,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:37,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:37,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:37,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:37,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:37,764 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:37,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,765 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,765 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,765 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:37,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316eb17f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:37,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:37,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,767 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:37,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:37,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:37,767 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,767 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:37,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:37,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,768 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:37,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:37,769 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:37,769 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:37,769 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:37,769 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:37,769 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:37,770 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,770 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,770 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,771 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:37,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:37,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,774 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,774 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:37,774 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:37,874 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:37,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:37,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:37,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:37,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:37,876 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:37,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:37,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:37,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,878 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,878 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:37,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:37,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:37,879 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,879 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:37,879 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:37,879 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:37,879 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:37,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,880 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,880 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,880 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:37,881 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:37,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b4bede[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:37,881 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,881 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:37,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:37,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:37,882 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,882 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:37,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:37,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,883 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,883 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:37,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:37,884 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:37,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:37,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:37,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:37,884 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:37,885 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,885 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,885 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,885 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,886 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:37,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:37,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:37,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,889 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:37,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:37,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:37,889 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:37,889 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:37,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803977964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803977966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803977967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:37,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803977967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,989 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:37,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:37,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:37,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:37,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:37,992 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:37,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:37,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:37,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,993 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:37,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:37,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:37,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:37,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:37,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:37,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:37,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:37,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:37,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:37,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:37,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:37,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:37,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:37,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,994 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:37,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:37,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:37,994 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:37,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:37,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:37,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:37,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:37,995 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,995 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:37,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:37,997 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:37,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e1acb9f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:37,997 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:37,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:37,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:37,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:37,998 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,998 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:37,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:37,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:37,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:37,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:37,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:37,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:37,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:37,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:37,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:38,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:38,000 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:38,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:38,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:38,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:38,000 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:38,001 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,001 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,001 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:38,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:38,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,004 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:38,004 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:38,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:38,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,104 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:38,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:38,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:38,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:38,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:38,108 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:38,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:38,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:38,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,109 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:38,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:38,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:38,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:38,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:38,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:38,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:38,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:38,110 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,110 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:38,111 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:38,111 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:38,111 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:38,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:38,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:38,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:38,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:38,113 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:38,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfb9457[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:38,113 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:38,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:38,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:38,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:38,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:38,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,114 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:38,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:38,116 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:38,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:38,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:38,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:38,116 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:38,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:38,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:38,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,121 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:38,121 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:38,121 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,121 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:38,121 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/b439f858eca345b9bca39894a283dbbb 2023-04-06 17:58:38,221 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:38,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:38,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:38,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:38,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:38,223 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:38,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:38,223 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:38,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,224 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,225 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,225 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:38,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:38,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:38,225 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:38,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:38,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:38,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:38,226 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:38,226 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:38,227 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,227 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,227 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,227 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:38,228 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:38,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395a37de[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:38,228 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,230 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:38,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:38,230 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:38,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:38,231 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:38,231 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,231 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,231 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:38,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:38,233 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:38,233 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:38,233 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:38,233 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:38,233 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:38,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,238 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:38,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:38,239 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:38,239 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:38,339 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:38,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:38,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:38,342 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:38,342 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,342 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:38,342 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:38,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,343 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,343 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,344 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,344 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:38,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:38,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:38,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:38,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:38,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:38,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:38,345 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,345 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:38,345 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:38,345 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:38,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:38,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:38,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:38,346 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,346 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,346 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,346 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:38,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7edf3c65[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:38,347 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:38,347 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,347 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:38,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:38,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:38,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:38,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:38,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:38,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:38,349 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:38,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:38,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:38,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:38,349 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:38,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:38,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:38,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,353 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:38,353 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:38,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,453 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:38,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:38,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:38,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:38,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:38,456 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:38,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:38,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:38,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,457 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:38,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:38,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:38,458 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,458 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:38,458 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:38,458 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:38,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:38,459 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,459 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,459 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,459 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:38,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:38,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e15612[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:38,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,461 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:38,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:38,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:38,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:38,462 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:38,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:38,463 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:38,464 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:38,464 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:38,464 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:38,464 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:38,464 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:38,465 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,465 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,465 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:38,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:38,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:38,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:38,468 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:38,468 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:38,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:38,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:38,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:38,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803978469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:38,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803978472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:38,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803978474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:38,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803978474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/7a8402b9a1ad4f4182c621a2ff3a7958 2023-04-06 17:58:38,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/81bfa87e01214cd8bf62399916fbee84 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81bfa87e01214cd8bf62399916fbee84 2023-04-06 17:58:38,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81bfa87e01214cd8bf62399916fbee84, entries=200, sequenceid=195, filesize=14.1 K 2023-04-06 17:58:38,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/b439f858eca345b9bca39894a283dbbb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b439f858eca345b9bca39894a283dbbb 2023-04-06 17:58:38,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b439f858eca345b9bca39894a283dbbb, entries=150, sequenceid=195, filesize=11.8 K 2023-04-06 17:58:38,552 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/7a8402b9a1ad4f4182c621a2ff3a7958 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7a8402b9a1ad4f4182c621a2ff3a7958 2023-04-06 17:58:38,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7a8402b9a1ad4f4182c621a2ff3a7958, entries=150, sequenceid=195, filesize=11.8 K 2023-04-06 17:58:38,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ec62d27e82d7188e5d149482edb0608d in 1234ms, sequenceid=195, compaction requested=true 2023-04-06 17:58:38,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:38,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:38,555 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:38,555 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:38,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:38,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:38,555 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38874 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:38,555 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:38,555 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:38,555 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:38,556 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,556 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,556 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/f0d4881c18dd4ad395b43f423e70294c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e64820b95f5f4c2f8bc7762788f49629, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81bfa87e01214cd8bf62399916fbee84] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=38.0 K 2023-04-06 17:58:38,556 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4abbfd26fb21430fb45f235c423b0a90, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b467a484ba9f4790bd30b01436d83949, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b439f858eca345b9bca39894a283dbbb] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=35.6 K 2023-04-06 17:58:38,556 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting f0d4881c18dd4ad395b43f423e70294c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680803915229 2023-04-06 17:58:38,556 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4abbfd26fb21430fb45f235c423b0a90, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680803915229 2023-04-06 17:58:38,556 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting e64820b95f5f4c2f8bc7762788f49629, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680803915559 2023-04-06 17:58:38,556 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b467a484ba9f4790bd30b01436d83949, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680803915559 2023-04-06 17:58:38,556 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 81bfa87e01214cd8bf62399916fbee84, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680803916689 2023-04-06 17:58:38,556 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b439f858eca345b9bca39894a283dbbb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680803916697 2023-04-06 17:58:38,561 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:38,563 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:38,568 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:38,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:38,570 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:38,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:38,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:38,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,571 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:38,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:38,571 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,571 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,572 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:38,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:38,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:38,572 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:38,573 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:38,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:38,577 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/314a4a5edb4e414781fc5e606acdc697 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/314a4a5edb4e414781fc5e606acdc697 2023-04-06 17:58:38,581 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into 314a4a5edb4e414781fc5e606acdc697(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:38,581 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:38,581 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=13, startTime=1680803918555; duration=0sec 2023-04-06 17:58:38,581 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:38,581 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:38,582 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:38,582 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:38,583 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:38,583 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e268ac4e0db45218dc92a6e63e418aa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e7c9a7d83d0b4f5b837abb6cd16d8929, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7a8402b9a1ad4f4182c621a2ff3a7958] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=35.6 K 2023-04-06 17:58:38,583 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3e268ac4e0db45218dc92a6e63e418aa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680803915229 2023-04-06 17:58:38,583 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e7c9a7d83d0b4f5b837abb6cd16d8929, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680803915559 2023-04-06 17:58:38,583 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7a8402b9a1ad4f4182c621a2ff3a7958, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680803916697 2023-04-06 17:58:38,591 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:38,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:38,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:38,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:38,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803978828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:38,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803978931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:38,975 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/de3c398098524b6db395dab9c746651e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/de3c398098524b6db395dab9c746651e 2023-04-06 17:58:38,978 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into de3c398098524b6db395dab9c746651e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:38,978 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:38,978 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=13, startTime=1680803918555; duration=0sec 2023-04-06 17:58:38,978 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:38,983 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/ad25f2bd5d4b43c8847eb4536ce869ef 2023-04-06 17:58:38,998 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/6fa09f9178244960904845e354fedcdd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6fa09f9178244960904845e354fedcdd 2023-04-06 17:58:39,001 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into 6fa09f9178244960904845e354fedcdd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:39,001 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:39,001 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=13, startTime=1680803918555; duration=0sec 2023-04-06 17:58:39,001 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:39,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803979133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,391 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/d9a0ff0756674ca4a440e73dfeec12ca 2023-04-06 17:58:39,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:39,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803979439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:39,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803979480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:39,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803979481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:39,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803979482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:39,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803979484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,802 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/e4d727e763994b81be840826a584463a 2023-04-06 17:58:39,805 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/ad25f2bd5d4b43c8847eb4536ce869ef as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad25f2bd5d4b43c8847eb4536ce869ef 2023-04-06 17:58:39,808 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad25f2bd5d4b43c8847eb4536ce869ef, entries=150, sequenceid=210, filesize=11.8 K 2023-04-06 17:58:39,808 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/d9a0ff0756674ca4a440e73dfeec12ca as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/d9a0ff0756674ca4a440e73dfeec12ca 2023-04-06 17:58:39,811 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/d9a0ff0756674ca4a440e73dfeec12ca, entries=150, sequenceid=210, filesize=11.8 K 2023-04-06 17:58:39,812 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/e4d727e763994b81be840826a584463a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e4d727e763994b81be840826a584463a 2023-04-06 17:58:39,815 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e4d727e763994b81be840826a584463a, entries=150, sequenceid=210, filesize=11.8 K 2023-04-06 17:58:39,815 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ec62d27e82d7188e5d149482edb0608d in 1242ms, sequenceid=210, compaction requested=false 2023-04-06 17:58:39,815 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:39,815 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:39,815 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:39,815 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:39,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:39,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:39,816 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,816 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:39,817 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,817 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:39,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:39,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:39,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@271933b5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:39,818 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,819 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,819 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:39,819 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:39,819 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:39,819 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,819 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:39,820 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:39,820 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,820 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:39,820 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:39,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:39,826 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:39,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:39,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:39,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:39,826 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:39,831 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,831 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:39,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,831 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:39,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:39,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:39,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:39,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:39,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:39,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:39,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:39,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,838 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:39,838 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:39,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:39,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:39,938 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:39,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:39,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:39,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:39,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:39,941 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:39,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:39,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:39,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:39,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:39,943 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,943 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:39,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:39,943 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:39,943 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:39,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:39,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:39,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:39,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:39,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:39,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:39,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:39,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:39,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:39,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:39,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:39,946 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:39,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:39,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:39,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:39,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:39,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803979985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:40,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803980088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:40,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803980293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,353 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/bbc1007e51a3461a8a5cf0376cb60d24 2023-04-06 17:58:40,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:40,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41112 deadline: 1680803980596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,762 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/44593d3454e3401ab1ac2bd9b0974ddb 2023-04-06 17:58:40,773 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/5e6d10ac9de144ad9dea4df78fe9ded6 2023-04-06 17:58:40,777 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/bbc1007e51a3461a8a5cf0376cb60d24 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/bbc1007e51a3461a8a5cf0376cb60d24 2023-04-06 17:58:40,782 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/bbc1007e51a3461a8a5cf0376cb60d24, entries=150, sequenceid=230, filesize=11.8 K 2023-04-06 17:58:40,782 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/44593d3454e3401ab1ac2bd9b0974ddb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/44593d3454e3401ab1ac2bd9b0974ddb 2023-04-06 17:58:40,785 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/44593d3454e3401ab1ac2bd9b0974ddb, entries=150, sequenceid=230, filesize=11.8 K 2023-04-06 17:58:40,786 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/5e6d10ac9de144ad9dea4df78fe9ded6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5e6d10ac9de144ad9dea4df78fe9ded6 2023-04-06 17:58:40,788 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5e6d10ac9de144ad9dea4df78fe9ded6, entries=150, sequenceid=230, filesize=11.8 K 2023-04-06 17:58:40,789 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ec62d27e82d7188e5d149482edb0608d in 843ms, sequenceid=230, compaction requested=true 2023-04-06 17:58:40,789 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:40,789 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:40,789 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:40,789 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:40,789 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,789 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:40,789 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:40,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,790 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:40,791 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:40,791 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:40,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:40,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f15144f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:40,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:40,792 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,792 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:40,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:40,792 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:40,793 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,793 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:40,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:40,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:40,793 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,793 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:40,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:40,795 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:40,795 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:40,795 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:40,795 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:40,795 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:40,797 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,797 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,797 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:40,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,797 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:40,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:40,798 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,800 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:40,800 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:40,800 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:40,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:40,800 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:40,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:40,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:40,800 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:40,801 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:40,801 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,801 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:40,801 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:40,801 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:40,801 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,801 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:40,901 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:40,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:40,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:40,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:40,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:40,904 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:40,904 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:40,904 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:40,904 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:40,904 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,904 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:40,905 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:40,905 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:40,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:40,905 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:40,905 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:40,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:40,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:40,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:40,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:40,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:40,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:40,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:40,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:40,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:40,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:40,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:40,906 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:40,906 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:40,906 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:40,915 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/ac399011994348388cd12b52b5a74122 2023-04-06 17:58:41,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:41,103 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:41,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803981161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803981267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,327 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/8c3e83144a33433d9a8c72f5af3db00e 2023-04-06 17:58:41,336 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/dca42489751a46f0bad56e11c82b3a8c 2023-04-06 17:58:41,338 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/ac399011994348388cd12b52b5a74122 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ac399011994348388cd12b52b5a74122 2023-04-06 17:58:41,341 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ac399011994348388cd12b52b5a74122, entries=150, sequenceid=245, filesize=11.8 K 2023-04-06 17:58:41,341 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/8c3e83144a33433d9a8c72f5af3db00e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/8c3e83144a33433d9a8c72f5af3db00e 2023-04-06 17:58:41,344 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/8c3e83144a33433d9a8c72f5af3db00e, entries=150, sequenceid=245, filesize=11.8 K 2023-04-06 17:58:41,345 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/dca42489751a46f0bad56e11c82b3a8c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/dca42489751a46f0bad56e11c82b3a8c 2023-04-06 17:58:41,347 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/dca42489751a46f0bad56e11c82b3a8c, entries=150, sequenceid=245, filesize=11.8 K 2023-04-06 17:58:41,348 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ec62d27e82d7188e5d149482edb0608d in 442ms, sequenceid=245, compaction requested=true 2023-04-06 17:58:41,348 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:41,348 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:41,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:41,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,349 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:41,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:41,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482e25dc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:41,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:41,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:41,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:41,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:41,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:41,351 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:41,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:41,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:41,353 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:41,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:41,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:41,353 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:41,353 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:41,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:41,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:41,356 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:41,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,357 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:41,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:41,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,357 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,357 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:41,357 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:41,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:41,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:41,457 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:41,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:41,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:41,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:41,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:41,460 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:41,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:41,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:41,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:41,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:41,461 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:41,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,461 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:41,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:41,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:41,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:41,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:41,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:41,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:41,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:41,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:41,462 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:41,462 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:41,462 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:41,462 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 17:58:41,462 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:41,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:41,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:41,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:41,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:41,463 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:41,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:41,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:41,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803981507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803981508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,520 DEBUG [Thread-3073] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:41,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803981514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803981515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803981518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803981621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803981621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803981623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803981630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803981826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803981826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803981830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:41,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803981838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,871 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/fc4b286df9834c30be95ccfa5a6334be 2023-04-06 17:58:41,886 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/057a332a3e974dd19c07ed29dcab482b 2023-04-06 17:58:41,894 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/a949687446f045db9d546bf2902bc8b1 2023-04-06 17:58:41,898 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/fc4b286df9834c30be95ccfa5a6334be as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/fc4b286df9834c30be95ccfa5a6334be 2023-04-06 17:58:41,901 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/fc4b286df9834c30be95ccfa5a6334be, entries=150, sequenceid=262, filesize=11.9 K 2023-04-06 17:58:41,902 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/057a332a3e974dd19c07ed29dcab482b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/057a332a3e974dd19c07ed29dcab482b 2023-04-06 17:58:41,905 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/057a332a3e974dd19c07ed29dcab482b, entries=150, sequenceid=262, filesize=11.9 K 2023-04-06 17:58:41,905 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/a949687446f045db9d546bf2902bc8b1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/a949687446f045db9d546bf2902bc8b1 2023-04-06 17:58:41,908 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/a949687446f045db9d546bf2902bc8b1, entries=150, sequenceid=262, filesize=11.9 K 2023-04-06 17:58:41,908 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ec62d27e82d7188e5d149482edb0608d in 446ms, sequenceid=262, compaction requested=true 2023-04-06 17:58:41,908 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:41,908 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:41,908 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:41,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:41,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:41,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:41,909 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:41,910 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,910 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:41,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:41,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f3ab0d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:41,911 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:41,911 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,911 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:41,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:41,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:41,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:41,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:41,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:41,912 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:41,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:41,915 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:41,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:41,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:41,915 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:41,915 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:41,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:41,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:41,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:41,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:41,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:41,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:41,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:41,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:41,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:41,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:41,920 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:41,920 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,920 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:41,920 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:41,920 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:41,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:41,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:41,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:41,920 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:41,920 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:42,020 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:42,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:42,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:42,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:42,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:42,023 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:42,023 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:42,023 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:42,023 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:42,023 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:42,023 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:42,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,024 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:42,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:42,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,024 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:42,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:42,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:42,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:42,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:42,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:42,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:42,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:42,025 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:42,026 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:42,025 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:42,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:42,026 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 17:58:42,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:42,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:42,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:42,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:42,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:42,026 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:42,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:42,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803982154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803982159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803982160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803982161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803982262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803982264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803982267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803982269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,437 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/10a25e6d6d004c6e838dbd0a341dc8c6 2023-04-06 17:58:42,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803982469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803982470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803982474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803982475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803982775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41060 deadline: 1680803982776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680803982779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:42,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41122 deadline: 1680803982783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:42,845 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/ce49560a092740a6b3b1359abbdd6379 2023-04-06 17:58:43,271 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/5d64894ce16549f1bbc72b54db8db36c 2023-04-06 17:58:43,274 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/10a25e6d6d004c6e838dbd0a341dc8c6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/10a25e6d6d004c6e838dbd0a341dc8c6 2023-04-06 17:58:43,277 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/10a25e6d6d004c6e838dbd0a341dc8c6, entries=150, sequenceid=277, filesize=11.9 K 2023-04-06 17:58:43,277 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/ce49560a092740a6b3b1359abbdd6379 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/ce49560a092740a6b3b1359abbdd6379 2023-04-06 17:58:43,280 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/ce49560a092740a6b3b1359abbdd6379, entries=150, sequenceid=277, filesize=11.9 K 2023-04-06 17:58:43,281 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/5d64894ce16549f1bbc72b54db8db36c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5d64894ce16549f1bbc72b54db8db36c 2023-04-06 17:58:43,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,283 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5d64894ce16549f1bbc72b54db8db36c, entries=150, sequenceid=277, filesize=11.9 K 2023-04-06 17:58:43,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803983279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,284 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ec62d27e82d7188e5d149482edb0608d in 1258ms, sequenceid=277, compaction requested=true 2023-04-06 17:58:43,284 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:43,284 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:43,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:43,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:43,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:43,286 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,286 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:43,287 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:43,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a99bcbd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:43,287 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,288 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:43,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:43,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,288 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:43,288 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:43,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:43,289 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,289 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:43,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,291 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:58:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:43,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:43,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:43,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:43,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:43,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:43,292 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:43,292 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:43,292 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:43,292 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:43,293 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:43,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:43,294 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,294 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,295 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,294 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,295 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:43,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:43,296 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,297 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:43,297 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:43,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:43,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/dcf17d1b5662413ca1de321ecdd1db0c 2023-04-06 17:58:43,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41060 deadline: 1680803983320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41122 deadline: 1680803983321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680803983329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,397 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:43,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:43,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:43,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:43,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:43,400 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:43,400 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,400 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:43,400 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:43,400 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,400 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,401 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,401 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:43,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:43,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:43,402 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:43,402 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,402 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:43,402 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:43,402 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:43,402 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,402 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:43,402 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:43,402 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:43,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:43,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:43,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:43,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,403 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:43,405 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:43,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d4972cd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:43,405 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,405 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:43,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:43,405 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,405 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:43,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:43,406 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:43,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:43,408 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:43,408 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:43,408 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:43,408 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:43,408 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:43,409 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,409 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,409 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,409 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,410 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,412 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,412 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,412 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:43,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:43,412 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:43,412 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,413 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:43,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,413 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:43,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41060 deadline: 1680803983429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41122 deadline: 1680803983430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680803983437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,513 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:43,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:43,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:43,516 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:43,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:43,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:43,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,516 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,517 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:43,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:43,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:43,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:43,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:43,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:43,518 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:43,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:43,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:43,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:43,519 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:43,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:43,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:43,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:43,520 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:43,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:43,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f0ae7d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:43,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,523 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:43,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:43,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:43,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:43,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:43,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:43,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:43,527 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:43,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:43,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:43,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:43,527 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:43,528 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,528 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,528 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,530 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:43,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,532 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:43,532 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:43,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,632 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:43,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:43,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:43,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:43,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:43,634 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:43,634 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:43,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:43,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,635 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,636 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,636 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,636 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:43,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:43,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:43,637 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:43,637 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,637 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:43,637 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:43,637 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:43,637 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,637 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:43,637 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:43,637 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,637 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:43,637 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:43,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:43,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:43,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,638 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:43,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:43,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d82c82d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:43,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,640 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:43,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:43,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:43,641 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,641 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:43,642 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41060 deadline: 1680803983637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41122 deadline: 1680803983637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:43,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:43,645 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:43,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:43,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:43,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:43,646 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:43,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,648 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:43,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:43,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:43,650 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:43,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,650 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:43,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680803983645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/7749191c3785437c9f75b66451daa636 2023-04-06 17:58:43,750 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:43,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:43,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:43,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:43,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:43,753 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:43,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:43,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:43,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,754 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:43,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:43,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:43,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:43,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:43,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:43,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:43,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:43,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:43,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:43,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:43,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:43,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:43,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,757 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:43,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a99f574[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:43,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:43,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:43,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:43,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:43,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:43,759 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,759 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:43,759 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,759 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:43,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:43,761 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:43,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:43,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:43,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:43,761 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:43,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:43,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:43,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:43,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,766 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,766 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:43,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,866 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:43,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:43,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:43,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:43,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:43,869 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:43,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:43,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:43,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,870 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:43,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:43,871 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,871 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:43,871 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:43,871 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:43,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:43,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:43,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cc029f0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:43,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:43,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,874 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:43,874 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:43,874 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:43,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:43,875 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,875 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,875 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,875 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:43,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:43,876 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:43,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:43,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:43,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:43,876 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:43,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,878 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,878 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:43,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:43,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:43,880 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,880 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:43,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,880 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:43,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41060 deadline: 1680803983945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41122 deadline: 1680803983946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:43,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680803983954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,980 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:43,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:43,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:43,983 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:43,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:43,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:43,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,983 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,984 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,984 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:43,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:43,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:43,985 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,985 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:43,985 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:43,985 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:43,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:43,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:43,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:43,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:43,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,986 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,987 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:43,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3db2e553[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:43,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,988 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:43,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:43,988 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,988 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:43,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:43,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:43,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:43,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:43,990 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:43,990 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:43,990 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:43,990 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:43,990 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:43,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:43,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,992 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:43,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:43,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:43,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:43,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:43,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:43,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:43,994 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:43,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:43,994 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:43,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,094 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:44,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:44,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:44,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:44,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:44,097 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:44,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:44,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:44,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,097 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,098 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,098 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:44,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:44,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:44,099 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:44,100 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:44,101 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:44,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca9168[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:44,101 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,102 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:44,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:44,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:44,102 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,102 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:44,103 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:44,103 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,103 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,103 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:44,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:44,104 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:44,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:44,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:44,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:44,104 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:44,105 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,105 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,105 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,106 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,112 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:44,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:44,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:44,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,112 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:44,112 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:44,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/c8ea5108671e48028ca76c0e00835e5d 2023-04-06 17:58:44,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/dcf17d1b5662413ca1de321ecdd1db0c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/dcf17d1b5662413ca1de321ecdd1db0c 2023-04-06 17:58:44,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/dcf17d1b5662413ca1de321ecdd1db0c, entries=250, sequenceid=296, filesize=16.7 K 2023-04-06 17:58:44,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/7749191c3785437c9f75b66451daa636 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/7749191c3785437c9f75b66451daa636 2023-04-06 17:58:44,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/7749191c3785437c9f75b66451daa636, entries=150, sequenceid=296, filesize=11.9 K 2023-04-06 17:58:44,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/c8ea5108671e48028ca76c0e00835e5d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/c8ea5108671e48028ca76c0e00835e5d 2023-04-06 17:58:44,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/c8ea5108671e48028ca76c0e00835e5d, entries=150, sequenceid=296, filesize=11.9 K 2023-04-06 17:58:44,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ec62d27e82d7188e5d149482edb0608d in 835ms, sequenceid=296, compaction requested=true 2023-04-06 17:58:44,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:44,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:44,126 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 17:58:44,126 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 17:58:44,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:44,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:44,132 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 90060 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 17:58:44,132 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85180 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 17:58:44,132 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:44,132 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:44,132 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,132 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,132 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/de3c398098524b6db395dab9c746651e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad25f2bd5d4b43c8847eb4536ce869ef, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/bbc1007e51a3461a8a5cf0376cb60d24, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ac399011994348388cd12b52b5a74122, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/fc4b286df9834c30be95ccfa5a6334be, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/10a25e6d6d004c6e838dbd0a341dc8c6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/dcf17d1b5662413ca1de321ecdd1db0c] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=87.9 K 2023-04-06 17:58:44,132 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/314a4a5edb4e414781fc5e606acdc697, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/d9a0ff0756674ca4a440e73dfeec12ca, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/44593d3454e3401ab1ac2bd9b0974ddb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/8c3e83144a33433d9a8c72f5af3db00e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/057a332a3e974dd19c07ed29dcab482b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/ce49560a092740a6b3b1359abbdd6379, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/7749191c3785437c9f75b66451daa636] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=83.2 K 2023-04-06 17:58:44,133 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting de3c398098524b6db395dab9c746651e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680803916697 2023-04-06 17:58:44,133 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 314a4a5edb4e414781fc5e606acdc697, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680803916697 2023-04-06 17:58:44,133 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ad25f2bd5d4b43c8847eb4536ce869ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680803917329 2023-04-06 17:58:44,133 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting d9a0ff0756674ca4a440e73dfeec12ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680803917329 2023-04-06 17:58:44,133 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting bbc1007e51a3461a8a5cf0376cb60d24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680803918813 2023-04-06 17:58:44,133 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 44593d3454e3401ab1ac2bd9b0974ddb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680803918813 2023-04-06 17:58:44,133 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ac399011994348388cd12b52b5a74122, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680803919964 2023-04-06 17:58:44,134 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 8c3e83144a33433d9a8c72f5af3db00e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680803919964 2023-04-06 17:58:44,134 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting fc4b286df9834c30be95ccfa5a6334be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680803921145 2023-04-06 17:58:44,134 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 057a332a3e974dd19c07ed29dcab482b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680803921145 2023-04-06 17:58:44,134 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 10a25e6d6d004c6e838dbd0a341dc8c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680803921506 2023-04-06 17:58:44,134 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting ce49560a092740a6b3b1359abbdd6379, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680803921506 2023-04-06 17:58:44,134 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting dcf17d1b5662413ca1de321ecdd1db0c, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680803922153 2023-04-06 17:58:44,134 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7749191c3785437c9f75b66451daa636, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680803922153 2023-04-06 17:58:44,144 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#460 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:44,145 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#461 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:44,154 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/b98ebd2297e64ab0ad3766972df8b3de as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b98ebd2297e64ab0ad3766972df8b3de 2023-04-06 17:58:44,157 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into b98ebd2297e64ab0ad3766972df8b3de(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:44,157 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:44,157 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=9, startTime=1680803924126; duration=0sec 2023-04-06 17:58:44,157 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:44,157 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 17:58:44,159 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85180 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 17:58:44,159 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:44,159 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,159 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6fa09f9178244960904845e354fedcdd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e4d727e763994b81be840826a584463a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5e6d10ac9de144ad9dea4df78fe9ded6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/dca42489751a46f0bad56e11c82b3a8c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/a949687446f045db9d546bf2902bc8b1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5d64894ce16549f1bbc72b54db8db36c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/c8ea5108671e48028ca76c0e00835e5d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=83.2 K 2023-04-06 17:58:44,159 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6fa09f9178244960904845e354fedcdd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680803916697 2023-04-06 17:58:44,159 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e4d727e763994b81be840826a584463a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680803917329 2023-04-06 17:58:44,159 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5e6d10ac9de144ad9dea4df78fe9ded6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680803918813 2023-04-06 17:58:44,160 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting dca42489751a46f0bad56e11c82b3a8c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680803919964 2023-04-06 17:58:44,160 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a949687446f045db9d546bf2902bc8b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680803921145 2023-04-06 17:58:44,160 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5d64894ce16549f1bbc72b54db8db36c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680803921506 2023-04-06 17:58:44,160 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c8ea5108671e48028ca76c0e00835e5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680803922153 2023-04-06 17:58:44,167 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#462 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:44,212 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:44,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:44,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:44,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:44,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:44,215 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:44,215 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,215 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:44,215 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:44,215 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,215 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,216 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:44,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:44,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:44,217 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:44,217 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,217 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:44,217 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:44,217 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:44,217 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:44,217 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:44,223 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/c9640bb975a9440bb838fcffd0d35da8 2023-04-06 17:58:44,231 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/3a700c91d3694ed389b5fa23e0c3e2fb 2023-04-06 17:58:44,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:44,295 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:44,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:44,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803984391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41060 deadline: 1680803984457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:44,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41122 deadline: 1680803984457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:44,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680803984462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:44,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803984502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,552 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/81b1975949ab4e25bff07aab00570940 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81b1975949ab4e25bff07aab00570940 2023-04-06 17:58:44,555 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into 81b1975949ab4e25bff07aab00570940(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:44,555 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:44,555 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=9, startTime=1680803924126; duration=0sec 2023-04-06 17:58:44,555 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:44,576 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/0447964ef8964048b1d256df1a20b562 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/0447964ef8964048b1d256df1a20b562 2023-04-06 17:58:44,579 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into 0447964ef8964048b1d256df1a20b562(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:44,579 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:44,579 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=9, startTime=1680803924126; duration=0sec 2023-04-06 17:58:44,579 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:44,639 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/2cdce89b22ec405ba6bd3162bfba6ce2 2023-04-06 17:58:44,642 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/c9640bb975a9440bb838fcffd0d35da8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/c9640bb975a9440bb838fcffd0d35da8 2023-04-06 17:58:44,644 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/c9640bb975a9440bb838fcffd0d35da8, entries=150, sequenceid=310, filesize=11.9 K 2023-04-06 17:58:44,644 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/3a700c91d3694ed389b5fa23e0c3e2fb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a700c91d3694ed389b5fa23e0c3e2fb 2023-04-06 17:58:44,647 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a700c91d3694ed389b5fa23e0c3e2fb, entries=150, sequenceid=310, filesize=11.9 K 2023-04-06 17:58:44,650 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/2cdce89b22ec405ba6bd3162bfba6ce2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2cdce89b22ec405ba6bd3162bfba6ce2 2023-04-06 17:58:44,653 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2cdce89b22ec405ba6bd3162bfba6ce2, entries=150, sequenceid=310, filesize=11.9 K 2023-04-06 17:58:44,653 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ec62d27e82d7188e5d149482edb0608d in 436ms, sequenceid=310, compaction requested=false 2023-04-06 17:58:44,653 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:44,653 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:44,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:44,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fdf690[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:44,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:44,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,656 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:44,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:44,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:44,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:44,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,657 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:44,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:44,659 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:44,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:44,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:44,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:44,659 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:44,660 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,660 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,660 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,661 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:44,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,663 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:44,663 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:44,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:58:44,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:44,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:44,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:44,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:44,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:44,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:44,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:44,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803984745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,763 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:44,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:44,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:44,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:44,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:44,766 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:44,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:44,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:44,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,767 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,767 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,767 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,767 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:44,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:44,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:44,773 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:44,773 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,773 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:44,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:44,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:44,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:44,781 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,781 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,781 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:44,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e3fd0b6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:44,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:44,782 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,783 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,783 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:44,783 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:44,783 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:44,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:44,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:44,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,784 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:44,786 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:44,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:44,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:44,786 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:44,786 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:44,786 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,786 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,787 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,787 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,788 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,790 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:44,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,790 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,790 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:44,790 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:44,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:44,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:44,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803984849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,890 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:44,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:44,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:44,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:44,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:44,893 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:44,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:44,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:44,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,894 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,894 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:44,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:44,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:44,895 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,895 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:44,895 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:44,895 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:44,895 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:44,896 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,896 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,896 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,896 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:44,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648d3c5f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:44,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:44,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:44,898 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,898 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:44,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:44,899 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,899 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:44,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:44,900 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:44,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:44,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:44,900 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:44,900 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:44,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,901 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:44,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,902 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:44,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:44,902 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:44,903 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,903 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,903 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:44,903 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:44,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:44,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:44,903 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:44,903 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:44,903 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:44,904 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:44,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:44,904 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:44,904 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:44,904 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:44,904 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:44,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,004 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,007 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,007 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,008 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,009 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,009 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,009 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,009 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,010 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,010 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,011 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@369ea077[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,011 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,012 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,012 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,012 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,012 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,012 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,013 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,013 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,013 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,013 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,015 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,015 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,015 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,015 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,015 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,016 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,018 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,019 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,019 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,020 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:45,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,020 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:45,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803985055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/1bc713ccad5e4b008f2034f00d164881 2023-04-06 17:58:45,120 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,122 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,123 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,125 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,125 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,125 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,125 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,126 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,126 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3b9942[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,128 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,128 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,129 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,130 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,130 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,132 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,133 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,135 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:45,135 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,135 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,235 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,238 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,238 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,239 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,240 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,240 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,240 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,240 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,241 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,241 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,241 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,241 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,241 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3158d3[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,244 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,246 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,246 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,247 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,247 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,247 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,247 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,249 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,249 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,349 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,352 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,356 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,356 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,356 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,356 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:45,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803985360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6753fe0e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,373 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,373 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,373 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,375 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,375 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,377 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,377 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,379 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,379 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:45,379 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:45,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41122 deadline: 1680803985471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:45,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41060 deadline: 1680803985471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:45,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680803985474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,479 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,482 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,482 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,482 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,482 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,482 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,482 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,483 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,485 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,485 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,485 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,485 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,485 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,486 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@240d778b[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,488 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,491 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,491 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,492 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,493 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,495 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,495 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,495 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/db303f4027084104a14ccecd9f572f0c 2023-04-06 17:58:45,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41088 deadline: 1680803985524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,527 DEBUG [Thread-3073] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:45,595 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,598 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,599 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,600 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,601 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,601 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,601 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,601 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,603 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22fa1bc1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,604 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,607 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,608 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:45,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,616 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,616 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,617 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,617 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,617 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,617 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,716 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,719 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,719 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,720 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,721 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,721 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,721 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,721 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42ed388d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,724 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,725 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,726 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,726 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,726 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,726 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,726 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,727 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,727 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,727 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,730 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,731 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,831 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,833 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,833 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,834 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,835 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,836 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,836 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:45,836 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:45,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:45,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ab61d50[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:45,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:45,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:45,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:45,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:45,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:45,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,839 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:45,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:45,841 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:45,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:45,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:45,841 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:45,841 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:45,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:45,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,842 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:45,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:45,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:45,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:45,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:45,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,845 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:45,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:45,845 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:45,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803985865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/fec80d83007f4bbc9ce99a14486d58bb 2023-04-06 17:58:45,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/1bc713ccad5e4b008f2034f00d164881 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/1bc713ccad5e4b008f2034f00d164881 2023-04-06 17:58:45,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/1bc713ccad5e4b008f2034f00d164881, entries=200, sequenceid=332, filesize=14.3 K 2023-04-06 17:58:45,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/db303f4027084104a14ccecd9f572f0c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/db303f4027084104a14ccecd9f572f0c 2023-04-06 17:58:45,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/db303f4027084104a14ccecd9f572f0c, entries=150, sequenceid=332, filesize=11.9 K 2023-04-06 17:58:45,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/fec80d83007f4bbc9ce99a14486d58bb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/fec80d83007f4bbc9ce99a14486d58bb 2023-04-06 17:58:45,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/fec80d83007f4bbc9ce99a14486d58bb, entries=150, sequenceid=332, filesize=11.9 K 2023-04-06 17:58:45,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ec62d27e82d7188e5d149482edb0608d in 1235ms, sequenceid=332, compaction requested=true 2023-04-06 17:58:45,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:45,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:45,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:45,943 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:45,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 17:58:45,943 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:45,944 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:45,944 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:45,944 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:45,944 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:45,944 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,944 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,944 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81b1975949ab4e25bff07aab00570940, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/c9640bb975a9440bb838fcffd0d35da8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/1bc713ccad5e4b008f2034f00d164881] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=38.8 K 2023-04-06 17:58:45,944 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b98ebd2297e64ab0ad3766972df8b3de, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a700c91d3694ed389b5fa23e0c3e2fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/db303f4027084104a14ccecd9f572f0c] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=36.4 K 2023-04-06 17:58:45,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 81b1975949ab4e25bff07aab00570940, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680803922153 2023-04-06 17:58:45,945 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b98ebd2297e64ab0ad3766972df8b3de, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680803922153 2023-04-06 17:58:45,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c9640bb975a9440bb838fcffd0d35da8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680803923311 2023-04-06 17:58:45,945 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 3a700c91d3694ed389b5fa23e0c3e2fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680803923311 2023-04-06 17:58:45,945 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 1bc713ccad5e4b008f2034f00d164881, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680803924362 2023-04-06 17:58:45,945 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting db303f4027084104a14ccecd9f572f0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680803924362 2023-04-06 17:58:45,945 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:45,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:45,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:45,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:45,948 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:45,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:45,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:45,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,948 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,949 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:45,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:45,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:45,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:45,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:45,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:45,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:45,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:45,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:45,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:45,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:45,950 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:45,950 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:45,950 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:45,963 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#469 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:45,963 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:45,978 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/19fe670834af43caa758a2804fb1346c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/19fe670834af43caa758a2804fb1346c 2023-04-06 17:58:45,981 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into 19fe670834af43caa758a2804fb1346c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:45,981 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:45,981 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=13, startTime=1680803925943; duration=0sec 2023-04-06 17:58:45,981 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:45,981 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:45,982 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:45,983 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:45,983 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:45,983 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/0447964ef8964048b1d256df1a20b562, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2cdce89b22ec405ba6bd3162bfba6ce2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/fec80d83007f4bbc9ce99a14486d58bb] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=36.4 K 2023-04-06 17:58:45,983 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 0447964ef8964048b1d256df1a20b562, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680803922153 2023-04-06 17:58:45,983 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 2cdce89b22ec405ba6bd3162bfba6ce2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680803923311 2023-04-06 17:58:45,983 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting fec80d83007f4bbc9ce99a14486d58bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680803924362 2023-04-06 17:58:45,987 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#472 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:46,374 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/b87bece5b2e64d4ba78b1e0cb25c3113 2023-04-06 17:58:46,379 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/f47534ce641e4256add6131fa1ce3650 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/f47534ce641e4256add6131fa1ce3650 2023-04-06 17:58:46,382 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into f47534ce641e4256add6131fa1ce3650(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:46,382 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:46,383 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=13, startTime=1680803925943; duration=0sec 2023-04-06 17:58:46,383 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:46,383 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/aa63c7ad71cd44bb92f3639a8d28a246 2023-04-06 17:58:46,392 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/7dadabd9bb354d4f83949614233546b7 2023-04-06 17:58:46,394 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/db40a0daabd546b0b93d9115cc349358 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/db40a0daabd546b0b93d9115cc349358 2023-04-06 17:58:46,397 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/b87bece5b2e64d4ba78b1e0cb25c3113 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/b87bece5b2e64d4ba78b1e0cb25c3113 2023-04-06 17:58:46,399 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into db40a0daabd546b0b93d9115cc349358(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:46,399 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:46,399 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=13, startTime=1680803925943; duration=0sec 2023-04-06 17:58:46,399 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:46,400 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/b87bece5b2e64d4ba78b1e0cb25c3113, entries=150, sequenceid=344, filesize=11.9 K 2023-04-06 17:58:46,401 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/aa63c7ad71cd44bb92f3639a8d28a246 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/aa63c7ad71cd44bb92f3639a8d28a246 2023-04-06 17:58:46,406 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/aa63c7ad71cd44bb92f3639a8d28a246, entries=150, sequenceid=344, filesize=11.9 K 2023-04-06 17:58:46,407 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/7dadabd9bb354d4f83949614233546b7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7dadabd9bb354d4f83949614233546b7 2023-04-06 17:58:46,410 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7dadabd9bb354d4f83949614233546b7, entries=150, sequenceid=344, filesize=11.9 K 2023-04-06 17:58:46,410 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for ec62d27e82d7188e5d149482edb0608d in 460ms, sequenceid=344, compaction requested=false 2023-04-06 17:58:46,410 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:46,410 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:46,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:46,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:46,410 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:46,411 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,411 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75c894a2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:46,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:46,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,414 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:46,414 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:46,414 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,414 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:46,414 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:46,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:46,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,415 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:46,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:46,417 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:46,417 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:46,417 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:46,417 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:46,417 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:46,418 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,418 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,418 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,418 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,419 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:46,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:46,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,421 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,421 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:46,421 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:46,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,421 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:46,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,521 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:46,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:46,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:46,523 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:46,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:46,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:46,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,524 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,524 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,524 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,524 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:46,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:46,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:46,525 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:46,525 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:46,525 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:46,525 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:46,525 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:46,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:46,526 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,526 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,527 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:46,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79eadc9[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:46,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:46,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,528 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,528 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:46,528 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:46,528 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:46,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:46,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:46,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:46,531 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:46,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:46,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:46,531 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:46,531 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:46,532 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,532 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,532 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,533 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:46,535 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:46,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:46,535 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,535 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:46,535 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,635 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:46,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:46,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:46,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:46,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:46,637 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:46,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:46,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:46,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,638 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:46,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:46,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:46,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:46,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:46,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:46,640 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:46,640 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,640 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:46,640 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:46,641 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:46,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:46,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:46,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:46,641 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,642 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:46,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c9cd4e7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:46,643 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:46,643 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,643 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:46,643 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:46,644 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,644 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:46,644 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:46,644 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,644 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,644 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,644 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:46,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:46,646 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:46,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:46,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:46,646 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:46,646 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:46,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,647 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,647 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,648 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:46,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:46,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,650 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:46,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,650 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:46,750 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:46,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:46,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:46,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:46,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:46,753 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:46,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:46,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:46,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,753 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,754 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:46,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:46,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:46,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:46,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:46,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:46,755 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:46,756 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,756 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:46,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ca10b9[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:46,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:46,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:46,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:46,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:46,758 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:46,760 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,760 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,760 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,760 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:46,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:46,761 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:46,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:46,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:46,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:46,761 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:46,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,763 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,764 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,765 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:46,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:46,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,765 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:46,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,765 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,766 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,866 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:46,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:46,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:46,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:46,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:46,868 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:46,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:46,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:46,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,868 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,869 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:46,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:46,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:46,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:46,870 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:46,870 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:46,871 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:46,871 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:46,872 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,872 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,873 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:46,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1796fd46[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:46,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:46,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,874 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,874 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:46,874 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:46,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:46,874 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:46,875 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,875 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:46,875 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,875 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:46,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:46,877 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:46,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:46,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:46,877 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:46,877 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:46,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,879 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,880 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:46,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:46,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,882 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,883 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:46,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,883 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,883 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,883 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,883 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:46,891 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:58:46,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:46,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:46,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:46,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:46,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:46,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803926921, max timestamp=1680803926956Num uniques 859; 2023-04-06 17:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803926921, max timestamp=1680803926956Num uniques 859; 2023-04-06 17:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803926921, max timestamp=1680803926956Num uniques 859; 2023-04-06 17:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:46,983 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:46,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:46,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:46,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:46,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:46,985 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:46,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:46,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:46,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,986 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,987 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:46,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:46,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:46,987 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,988 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:46,988 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:46,988 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:46,989 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:46,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6594ea98[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:46,990 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:46,990 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:46,991 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,991 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:46,991 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,992 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:46,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:46,993 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:46,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:46,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:46,993 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:46,993 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:46,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,994 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:46,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:46,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,997 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:46,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:46,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:46,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:46,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:46,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:46,997 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:46,997 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:46,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:47,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803987036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,097 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:47,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:47,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:47,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:47,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:47,099 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:47,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:47,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:47,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,100 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,101 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,101 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,101 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:47,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:47,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:47,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:47,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:47,102 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:47,103 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:47,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f652121[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:47,104 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,105 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:47,105 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,105 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:47,105 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,106 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:47,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:47,107 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:47,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:47,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:47,108 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:47,108 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:47,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,108 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:47,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:47,112 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,112 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,113 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:47,113 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:47,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:47,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803987137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,213 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:47,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:47,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:47,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:47,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:47,216 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:47,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:47,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:47,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,216 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,217 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,217 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:47,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:47,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:47,219 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,219 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,219 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,219 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:47,220 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@595cb3ac[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:47,220 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,221 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,221 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,221 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:47,221 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:47,221 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:47,221 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:47,222 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:47,223 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,222 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,223 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:47,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:47,224 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:47,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:47,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:47,224 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:47,224 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:47,225 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,225 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,225 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,225 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,227 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,228 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,228 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,228 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,228 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,228 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:47,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:47,228 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,229 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,229 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,229 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,228 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,229 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:47,229 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:47,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/e647d5cddac141688e23a2e96183121d 2023-04-06 17:58:47,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/6ada11537fa341f996957315f313e51e 2023-04-06 17:58:47,329 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:47,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:47,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:47,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:47,331 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:47,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:47,331 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:47,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,332 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,332 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:47,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:47,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:47,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:47,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:47,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:47,334 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:47,335 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,335 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,335 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,335 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:47,336 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:47,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7980090c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:47,336 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,337 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,337 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:47,337 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,337 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:47,337 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:47,337 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:47,337 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,337 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,338 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:47,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:47,339 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:47,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:47,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:47,339 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:47,339 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:47,340 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,341 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,341 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,344 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:47,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803987341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,346 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:47,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:47,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,355 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:47,354 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,355 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:47,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,455 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:47,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:47,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:47,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:47,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:47,457 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:47,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,457 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:47,458 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:47,458 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,458 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,458 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,458 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,459 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:47,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:47,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:47,460 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,460 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:47,460 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:47,460 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:47,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:47,461 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:47,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:47,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b02fc8e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:47,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,463 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:47,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:47,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:47,463 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:47,464 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,464 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:47,464 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,464 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:47,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:47,466 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:47,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:47,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:47,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:47,467 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:47,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,468 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,471 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:47,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,471 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:47,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:47,471 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,471 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41122 deadline: 1680803987483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,487 DEBUG [Thread-3077] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:47,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:47,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680803987485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,492 DEBUG [Thread-3071] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:47,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:47,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41060 deadline: 1680803987490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,495 DEBUG [Thread-3069] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:58:47,571 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:47,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:47,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:47,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:47,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:47,574 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:47,574 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,574 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:47,574 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:47,576 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,576 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,577 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,577 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,577 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,577 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:47,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:47,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:47,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:47,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:47,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:47,578 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,578 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:47,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:47,578 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:47,578 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:47,578 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:47,579 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,579 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:47,579 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:47,579 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,579 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,580 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:47,581 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:47,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d68aee6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:47,581 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,581 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,581 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:47,581 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:47,581 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,582 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:47,581 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:47,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,582 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:47,582 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,582 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:47,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:47,584 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:47,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:47,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:47,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:47,584 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:47,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,585 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,586 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,587 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,588 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:47,589 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:47,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:47,589 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:47,589 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:47,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803987646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,689 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:47,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:47,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:47,692 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:47,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:47,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:47,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,692 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,693 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,693 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,693 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,693 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:47,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:47,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:47,694 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,694 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:47,694 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:47,694 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:47,695 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:47,695 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,695 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,695 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,695 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:47,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:47,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@115ab34[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:47,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,697 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,697 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:47,697 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,697 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:47,697 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:47,697 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:47,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:47,698 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:47,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:47,700 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:47,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:47,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:47,700 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:47,700 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:47,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:47,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,701 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,702 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,703 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,703 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,703 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,703 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:47,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:47,703 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:47,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,703 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:47,703 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,704 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:47,704 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,704 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:47,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:47,704 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:47,704 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/b97d254e1a0e4351b775b37ce1fa9649 2023-04-06 17:58:47,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/e647d5cddac141688e23a2e96183121d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e647d5cddac141688e23a2e96183121d 2023-04-06 17:58:47,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e647d5cddac141688e23a2e96183121d, entries=200, sequenceid=361, filesize=14.3 K 2023-04-06 17:58:47,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/6ada11537fa341f996957315f313e51e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/6ada11537fa341f996957315f313e51e 2023-04-06 17:58:47,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/6ada11537fa341f996957315f313e51e, entries=150, sequenceid=361, filesize=11.9 K 2023-04-06 17:58:47,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/b97d254e1a0e4351b775b37ce1fa9649 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b97d254e1a0e4351b775b37ce1fa9649 2023-04-06 17:58:47,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b97d254e1a0e4351b775b37ce1fa9649, entries=150, sequenceid=361, filesize=11.9 K 2023-04-06 17:58:47,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ec62d27e82d7188e5d149482edb0608d in 837ms, sequenceid=361, compaction requested=true 2023-04-06 17:58:47,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:47,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:47,728 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:47,728 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:47,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:47,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:47,729 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39816 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:47,729 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:47,729 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,729 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/19fe670834af43caa758a2804fb1346c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/b87bece5b2e64d4ba78b1e0cb25c3113, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e647d5cddac141688e23a2e96183121d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=38.9 K 2023-04-06 17:58:47,730 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:47,730 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 19fe670834af43caa758a2804fb1346c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680803924362 2023-04-06 17:58:47,730 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:47,730 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,730 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/f47534ce641e4256add6131fa1ce3650, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/aa63c7ad71cd44bb92f3639a8d28a246, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/6ada11537fa341f996957315f313e51e] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=36.5 K 2023-04-06 17:58:47,730 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b87bece5b2e64d4ba78b1e0cb25c3113, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1680803924715 2023-04-06 17:58:47,730 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f47534ce641e4256add6131fa1ce3650, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680803924362 2023-04-06 17:58:47,731 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting e647d5cddac141688e23a2e96183121d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680803926880 2023-04-06 17:58:47,731 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting aa63c7ad71cd44bb92f3639a8d28a246, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1680803924715 2023-04-06 17:58:47,731 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6ada11537fa341f996957315f313e51e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680803926880 2023-04-06 17:58:47,735 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#479 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:47,736 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#478 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:47,804 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:47,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:47,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:47,806 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:47,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:47,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:47,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:47,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:47,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:47,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:47,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:47,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:47,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:47,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:47,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:47,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:47,809 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:47,809 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:58:47,815 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/90ee65981ea848319d69122465096483 2023-04-06 17:58:48,142 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/0853b9e7d69a425fabb23397c9afd15d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0853b9e7d69a425fabb23397c9afd15d 2023-04-06 17:58:48,142 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/94ec6d3bd71746db8b969619becd237e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/94ec6d3bd71746db8b969619becd237e 2023-04-06 17:58:48,146 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into 0853b9e7d69a425fabb23397c9afd15d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:48,146 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:48,146 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=13, startTime=1680803927728; duration=0sec 2023-04-06 17:58:48,146 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:48,146 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:58:48,147 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into 94ec6d3bd71746db8b969619becd237e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:48,147 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:48,147 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=13, startTime=1680803927728; duration=0sec 2023-04-06 17:58:48,147 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:48,147 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:58:48,147 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:48,147 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:48,147 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/db40a0daabd546b0b93d9115cc349358, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7dadabd9bb354d4f83949614233546b7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b97d254e1a0e4351b775b37ce1fa9649] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=36.5 K 2023-04-06 17:58:48,148 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting db40a0daabd546b0b93d9115cc349358, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680803924362 2023-04-06 17:58:48,148 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 7dadabd9bb354d4f83949614233546b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1680803924715 2023-04-06 17:58:48,148 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b97d254e1a0e4351b775b37ce1fa9649, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680803926880 2023-04-06 17:58:48,152 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:48,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:48,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:48,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803988186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,227 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/c35d7c9234eb43a7889468245e641d0a 2023-04-06 17:58:48,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:48,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803988291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803988497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,558 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/3a378637adae470c820b8e463bcbf04c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3a378637adae470c820b8e463bcbf04c 2023-04-06 17:58:48,561 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into 3a378637adae470c820b8e463bcbf04c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:48,561 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:48,561 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=13, startTime=1680803927729; duration=0sec 2023-04-06 17:58:48,561 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:48,635 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/3e946dbbb141432da6baf1ae14861aff 2023-04-06 17:58:48,638 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/90ee65981ea848319d69122465096483 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90ee65981ea848319d69122465096483 2023-04-06 17:58:48,640 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90ee65981ea848319d69122465096483, entries=150, sequenceid=398, filesize=11.9 K 2023-04-06 17:58:48,641 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/c35d7c9234eb43a7889468245e641d0a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c35d7c9234eb43a7889468245e641d0a 2023-04-06 17:58:48,644 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c35d7c9234eb43a7889468245e641d0a, entries=150, sequenceid=398, filesize=11.9 K 2023-04-06 17:58:48,644 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/3e946dbbb141432da6baf1ae14861aff as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e946dbbb141432da6baf1ae14861aff 2023-04-06 17:58:48,647 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e946dbbb141432da6baf1ae14861aff, entries=150, sequenceid=398, filesize=11.9 K 2023-04-06 17:58:48,647 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for ec62d27e82d7188e5d149482edb0608d in 838ms, sequenceid=398, compaction requested=false 2023-04-06 17:58:48,647 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:48,648 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:48,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:48,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:48,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:48,648 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:48,649 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,649 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:48,649 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,649 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:48,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:48,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78240a73[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:48,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:48,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,651 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:48,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:48,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:48,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:48,651 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:48,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:48,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:48,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:48,653 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:48,653 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:48,653 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:48,653 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:48,653 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:48,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:48,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,654 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:48,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:48,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,656 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:48,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:48,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:48,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:48,657 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:48,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:48,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:48,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:48,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:48,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,658 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:48,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:58:48,658 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:48,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:48,758 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:48,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:48,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:58:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:58:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:58:48,760 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:58:48,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:48,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:58:48,760 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:58:48,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:48,762 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:48,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:48,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:48,762 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:58:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:48,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:58:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:48,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:58:48,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:58:48,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:58:48,763 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:58:48,763 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:58:48,763 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:58:48,763 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:58:48,763 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:58:48,763 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:48,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. started... 2023-04-06 17:58:48,764 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:58:48,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:48,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:48,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:48,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:48,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:48,764 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:48,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:48,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. as already flushing 2023-04-06 17:58:48,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803928847, max timestamp=1680803928863Num uniques 859; 2023-04-06 17:58:48,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:48,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803928847, max timestamp=1680803928863Num uniques 859; 2023-04-06 17:58:48,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803928847, max timestamp=1680803928863Num uniques 859; 2023-04-06 17:58:48,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:48,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:58:48,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803988970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803989076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,137 DEBUG [Thread-3088] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c9c1feb to 127.0.0.1:49384 2023-04-06 17:58:49,137 DEBUG [Thread-3088] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:49,137 DEBUG [Thread-3086] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x519db5f2 to 127.0.0.1:49384 2023-04-06 17:58:49,137 DEBUG [Thread-3086] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:49,140 DEBUG [Thread-3082] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e7a5e9a to 127.0.0.1:49384 2023-04-06 17:58:49,140 DEBUG [Thread-3082] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:49,143 DEBUG [Thread-3080] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4048dcb4 to 127.0.0.1:49384 2023-04-06 17:58:49,143 DEBUG [Thread-3080] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:49,143 DEBUG [Thread-3084] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62447900 to 127.0.0.1:49384 2023-04-06 17:58:49,143 DEBUG [Thread-3084] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:49,169 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/11b180ed85154e76888aa25821989cc3 2023-04-06 17:58:49,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803989281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,581 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/e1b146dd0ef74164a7beb64a70037782 2023-04-06 17:58:49,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:58:49,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41112 deadline: 1680803989586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,599 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/6a9f20fe63fb404185bbbece7ac3a398 2023-04-06 17:58:49,604 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/11b180ed85154e76888aa25821989cc3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/11b180ed85154e76888aa25821989cc3 2023-04-06 17:58:49,608 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/11b180ed85154e76888aa25821989cc3, entries=150, sequenceid=411, filesize=11.9 K 2023-04-06 17:58:49,609 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/e1b146dd0ef74164a7beb64a70037782 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/e1b146dd0ef74164a7beb64a70037782 2023-04-06 17:58:49,613 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/e1b146dd0ef74164a7beb64a70037782, entries=150, sequenceid=411, filesize=11.9 K 2023-04-06 17:58:49,614 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/6a9f20fe63fb404185bbbece7ac3a398 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6a9f20fe63fb404185bbbece7ac3a398 2023-04-06 17:58:49,618 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6a9f20fe63fb404185bbbece7ac3a398, entries=150, sequenceid=411, filesize=11.9 K 2023-04-06 17:58:49,619 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ec62d27e82d7188e5d149482edb0608d in 855ms, sequenceid=411, compaction requested=true 2023-04-06 17:58:49,619 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:49,619 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:49,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:58:49,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:58:49,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:58:49,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:58:49,621 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:49,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:58:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:49,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:58:49,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@546e7f75[Count = 0] remaining members to acquire global barrier 2023-04-06 17:58:49,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:58:49,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,624 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,624 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:58:49,624 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:58:49,624 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:58:49,624 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,624 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:58:49,625 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:58:49,625 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,625 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:49,625 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:58:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:58:49,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:58:49,628 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:58:49,628 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:58:49,628 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:58:49,628 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:58:49,628 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:58:49,629 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:49,629 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:58:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:58:49,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:49,629 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:58:49,630 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:58:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,631 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:58:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:58:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,633 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,633 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:58:49,633 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:49,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:58:49,634 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:58:49,633 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:58:49,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:58:49,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:58:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:58:49,634 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:58:49,634 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:49,634 DEBUG [Thread-3079] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:58:49,634 DEBUG [Thread-3079] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:58:49,634 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:58:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:58:49,634 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:58:49,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:58:49,734 DEBUG [Thread-3079] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:58:49,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:58:50,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:50,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-06 17:58:50,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:50,091 DEBUG [Thread-3075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7cb49278 to 127.0.0.1:49384 2023-04-06 17:58:50,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:58:50,092 DEBUG [Thread-3075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:50,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:50,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:58:50,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:50,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:58:50,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/802af9dc4247476c8c957d602d33234b 2023-04-06 17:58:50,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/4084934601d643ec984b9a747ccfc1b7 2023-04-06 17:58:50,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/2602c3304bef431fa3cc3c4048ce521d 2023-04-06 17:58:50,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/802af9dc4247476c8c957d602d33234b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/802af9dc4247476c8c957d602d33234b 2023-04-06 17:58:50,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/802af9dc4247476c8c957d602d33234b, entries=150, sequenceid=449, filesize=11.9 K 2023-04-06 17:58:50,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/4084934601d643ec984b9a747ccfc1b7 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4084934601d643ec984b9a747ccfc1b7 2023-04-06 17:58:50,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4084934601d643ec984b9a747ccfc1b7, entries=150, sequenceid=449, filesize=11.9 K 2023-04-06 17:58:50,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/2602c3304bef431fa3cc3c4048ce521d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2602c3304bef431fa3cc3c4048ce521d 2023-04-06 17:58:50,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2602c3304bef431fa3cc3c4048ce521d, entries=150, sequenceid=449, filesize=11.9 K 2023-04-06 17:58:50,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=0 B/0 for ec62d27e82d7188e5d149482edb0608d in 851ms, sequenceid=449, compaction requested=true 2023-04-06 17:58:50,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:50,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:50,942 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:50,942 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:50,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:50,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:50,943 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:50,943 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:50,943 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/A is initiating minor compaction (all files) 2023-04-06 17:58:50,943 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/B is initiating minor compaction (all files) 2023-04-06 17:58:50,943 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/A in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:50,943 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/B in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:50,943 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0853b9e7d69a425fabb23397c9afd15d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90ee65981ea848319d69122465096483, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/11b180ed85154e76888aa25821989cc3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/802af9dc4247476c8c957d602d33234b] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=48.5 K 2023-04-06 17:58:50,943 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/94ec6d3bd71746db8b969619becd237e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c35d7c9234eb43a7889468245e641d0a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/e1b146dd0ef74164a7beb64a70037782, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4084934601d643ec984b9a747ccfc1b7] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=48.5 K 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 0853b9e7d69a425fabb23397c9afd15d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680803926880 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 94ec6d3bd71746db8b969619becd237e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680803926880 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 90ee65981ea848319d69122465096483, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680803927005 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c35d7c9234eb43a7889468245e641d0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680803927005 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 11b180ed85154e76888aa25821989cc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680803928157 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e1b146dd0ef74164a7beb64a70037782, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680803928157 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 802af9dc4247476c8c957d602d33234b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680803928933 2023-04-06 17:58:50,944 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4084934601d643ec984b9a747ccfc1b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680803928933 2023-04-06 17:58:50,950 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#A#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:50,951 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#B#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:51,364 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/621316f148b94801b750b9d68cab4d42 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/621316f148b94801b750b9d68cab4d42 2023-04-06 17:58:51,365 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/73d67aa9c8e347d89287cf87e17f7354 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/73d67aa9c8e347d89287cf87e17f7354 2023-04-06 17:58:51,368 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/A of ec62d27e82d7188e5d149482edb0608d into 621316f148b94801b750b9d68cab4d42(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:51,368 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/B of ec62d27e82d7188e5d149482edb0608d into 73d67aa9c8e347d89287cf87e17f7354(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:51,369 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:51,369 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:51,369 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/A, priority=12, startTime=1680803930942; duration=0sec 2023-04-06 17:58:51,369 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/B, priority=12, startTime=1680803930942; duration=0sec 2023-04-06 17:58:51,369 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:51,369 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:58:51,369 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:58:51,370 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:58:51,370 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): ec62d27e82d7188e5d149482edb0608d/C is initiating minor compaction (all files) 2023-04-06 17:58:51,370 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec62d27e82d7188e5d149482edb0608d/C in TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:51,370 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3a378637adae470c820b8e463bcbf04c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e946dbbb141432da6baf1ae14861aff, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6a9f20fe63fb404185bbbece7ac3a398, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2602c3304bef431fa3cc3c4048ce521d] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp, totalSize=48.5 K 2023-04-06 17:58:51,370 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3a378637adae470c820b8e463bcbf04c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680803926880 2023-04-06 17:58:51,371 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3e946dbbb141432da6baf1ae14861aff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680803927005 2023-04-06 17:58:51,371 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6a9f20fe63fb404185bbbece7ac3a398, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680803928157 2023-04-06 17:58:51,371 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 2602c3304bef431fa3cc3c4048ce521d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680803928933 2023-04-06 17:58:51,378 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec62d27e82d7188e5d149482edb0608d#C#compaction#492 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:58:51,497 DEBUG [Thread-3071] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17a38ae8 to 127.0.0.1:49384 2023-04-06 17:58:51,497 DEBUG [Thread-3071] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:51,507 DEBUG [Thread-3069] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4391d6f4 to 127.0.0.1:49384 2023-04-06 17:58:51,507 DEBUG [Thread-3077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50e89dd2 to 127.0.0.1:49384 2023-04-06 17:58:51,507 DEBUG [Thread-3077] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:51,507 DEBUG [Thread-3069] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:51,790 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/5ee5cbb4cfd342f58a96107e5eb88d40 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5ee5cbb4cfd342f58a96107e5eb88d40 2023-04-06 17:58:51,794 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec62d27e82d7188e5d149482edb0608d/C of ec62d27e82d7188e5d149482edb0608d into 5ee5cbb4cfd342f58a96107e5eb88d40(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:58:51,794 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:51,794 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d., storeName=ec62d27e82d7188e5d149482edb0608d/C, priority=12, startTime=1680803930942; duration=0sec 2023-04-06 17:58:51,794 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:58:55,549 DEBUG [Thread-3073] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e21fa39 to 127.0.0.1:49384 2023-04-06 17:58:55,550 DEBUG [Thread-3073] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:55,550 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 17:58:55,550 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-04-06 17:58:55,550 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 185 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 43 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2438 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 7310 rows 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2451 2023-04-06 17:58:55,551 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 7346 rows 2023-04-06 17:58:55,552 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2450 2023-04-06 17:58:55,552 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 7341 rows 2023-04-06 17:58:55,552 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2440 2023-04-06 17:58:55,552 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 7315 rows 2023-04-06 17:58:55,552 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(402): scanned 2435 2023-04-06 17:58:55,552 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(403): verified 7302 rows 2023-04-06 17:58:55,552 INFO [Listener at localhost.localdomain/40169] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 17:58:55,552 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x796f86e5 to 127.0.0.1:49384 2023-04-06 17:58:55,553 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:58:55,555 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 17:58:55,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 17:58:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:55,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 17:58:55,560 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803935560"}]},"ts":"1680803935560"} 2023-04-06 17:58:55,561 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 17:58:55,571 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 17:58:55,571 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec62d27e82d7188e5d149482edb0608d, UNASSIGN}] 2023-04-06 17:58:55,572 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec62d27e82d7188e5d149482edb0608d, UNASSIGN 2023-04-06 17:58:55,573 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ec62d27e82d7188e5d149482edb0608d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:55,573 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803935573"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803935573"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803935573"}]},"ts":"1680803935573"} 2023-04-06 17:58:55,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:58:55,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 17:58:55,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:55,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ec62d27e82d7188e5d149482edb0608d, disabling compactions & flushes 2023-04-06 17:58:55,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:55,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:55,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. after waiting 0 ms 2023-04-06 17:58:55,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:55,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing ec62d27e82d7188e5d149482edb0608d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 17:58:55,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=A 2023-04-06 17:58:55,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:55,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=B 2023-04-06 17:58:55,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:55,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec62d27e82d7188e5d149482edb0608d, store=C 2023-04-06 17:58:55,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:58:55,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 17:58:56,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/90381db0f7364206b5592ab0a8146a5e 2023-04-06 17:58:56,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 17:58:56,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/76d2ab9c222d4e42959ef87af778e2e6 2023-04-06 17:58:56,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 17:58:56,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/abfc2c668fc443968a8bf471ba48edef 2023-04-06 17:58:56,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/A/90381db0f7364206b5592ab0a8146a5e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90381db0f7364206b5592ab0a8146a5e 2023-04-06 17:58:56,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90381db0f7364206b5592ab0a8146a5e, entries=150, sequenceid=459, filesize=11.9 K 2023-04-06 17:58:56,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/B/76d2ab9c222d4e42959ef87af778e2e6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/76d2ab9c222d4e42959ef87af778e2e6 2023-04-06 17:58:56,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/76d2ab9c222d4e42959ef87af778e2e6, entries=150, sequenceid=459, filesize=11.9 K 2023-04-06 17:58:56,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/.tmp/C/abfc2c668fc443968a8bf471ba48edef as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/abfc2c668fc443968a8bf471ba48edef 2023-04-06 17:58:56,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/abfc2c668fc443968a8bf471ba48edef, entries=150, sequenceid=459, filesize=11.9 K 2023-04-06 17:58:56,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for ec62d27e82d7188e5d149482edb0608d in 1270ms, sequenceid=459, compaction requested=false 2023-04-06 17:58:57,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad504c1fb8004530955d651136f15bd6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/75a242fac00f423f8e614834a9819b80, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d30531b7a04004ba06fd3845b27992, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/af77859c16da45678c2d6de83c6c1e19, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d183ed3c614b218eb13817af3e26d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/41cc90541de8454b9fb9235f3caab84c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/7abc6ccf15f0465bb05bfac8166aefb8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/8eb5630bf03b4e618dcc00646c536242, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/aae5893adf974d6592027994d596f536, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0ec976034a3e4066b2ac7257775c709b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/f0d4881c18dd4ad395b43f423e70294c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e64820b95f5f4c2f8bc7762788f49629, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81bfa87e01214cd8bf62399916fbee84, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/de3c398098524b6db395dab9c746651e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad25f2bd5d4b43c8847eb4536ce869ef, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/bbc1007e51a3461a8a5cf0376cb60d24, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ac399011994348388cd12b52b5a74122, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/fc4b286df9834c30be95ccfa5a6334be, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/10a25e6d6d004c6e838dbd0a341dc8c6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/dcf17d1b5662413ca1de321ecdd1db0c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81b1975949ab4e25bff07aab00570940, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/c9640bb975a9440bb838fcffd0d35da8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/1bc713ccad5e4b008f2034f00d164881, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/19fe670834af43caa758a2804fb1346c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/b87bece5b2e64d4ba78b1e0cb25c3113, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e647d5cddac141688e23a2e96183121d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0853b9e7d69a425fabb23397c9afd15d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90ee65981ea848319d69122465096483, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/11b180ed85154e76888aa25821989cc3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/802af9dc4247476c8c957d602d33234b] to archive 2023-04-06 17:58:57,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:57,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad504c1fb8004530955d651136f15bd6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad504c1fb8004530955d651136f15bd6 2023-04-06 17:58:57,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/75a242fac00f423f8e614834a9819b80 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/75a242fac00f423f8e614834a9819b80 2023-04-06 17:58:57,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d30531b7a04004ba06fd3845b27992 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d30531b7a04004ba06fd3845b27992 2023-04-06 17:58:57,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/af77859c16da45678c2d6de83c6c1e19 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/af77859c16da45678c2d6de83c6c1e19 2023-04-06 17:58:57,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d183ed3c614b218eb13817af3e26d2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/48d183ed3c614b218eb13817af3e26d2 2023-04-06 17:58:57,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/41cc90541de8454b9fb9235f3caab84c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/41cc90541de8454b9fb9235f3caab84c 2023-04-06 17:58:57,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/7abc6ccf15f0465bb05bfac8166aefb8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/7abc6ccf15f0465bb05bfac8166aefb8 2023-04-06 17:58:57,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/8eb5630bf03b4e618dcc00646c536242 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/8eb5630bf03b4e618dcc00646c536242 2023-04-06 17:58:57,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/aae5893adf974d6592027994d596f536 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/aae5893adf974d6592027994d596f536 2023-04-06 17:58:57,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/57cc4ec5a1cb4e05a0c08dde6d5d3d7e 2023-04-06 17:58:57,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0ec976034a3e4066b2ac7257775c709b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0ec976034a3e4066b2ac7257775c709b 2023-04-06 17:58:57,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/f0d4881c18dd4ad395b43f423e70294c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/f0d4881c18dd4ad395b43f423e70294c 2023-04-06 17:58:57,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e64820b95f5f4c2f8bc7762788f49629 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e64820b95f5f4c2f8bc7762788f49629 2023-04-06 17:58:57,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81bfa87e01214cd8bf62399916fbee84 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81bfa87e01214cd8bf62399916fbee84 2023-04-06 17:58:57,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/de3c398098524b6db395dab9c746651e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/de3c398098524b6db395dab9c746651e 2023-04-06 17:58:57,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad25f2bd5d4b43c8847eb4536ce869ef to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ad25f2bd5d4b43c8847eb4536ce869ef 2023-04-06 17:58:57,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/bbc1007e51a3461a8a5cf0376cb60d24 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/bbc1007e51a3461a8a5cf0376cb60d24 2023-04-06 17:58:57,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ac399011994348388cd12b52b5a74122 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/ac399011994348388cd12b52b5a74122 2023-04-06 17:58:57,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/fc4b286df9834c30be95ccfa5a6334be to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/fc4b286df9834c30be95ccfa5a6334be 2023-04-06 17:58:57,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/10a25e6d6d004c6e838dbd0a341dc8c6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/10a25e6d6d004c6e838dbd0a341dc8c6 2023-04-06 17:58:57,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/dcf17d1b5662413ca1de321ecdd1db0c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/dcf17d1b5662413ca1de321ecdd1db0c 2023-04-06 17:58:57,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81b1975949ab4e25bff07aab00570940 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/81b1975949ab4e25bff07aab00570940 2023-04-06 17:58:57,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/c9640bb975a9440bb838fcffd0d35da8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/c9640bb975a9440bb838fcffd0d35da8 2023-04-06 17:58:57,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/1bc713ccad5e4b008f2034f00d164881 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/1bc713ccad5e4b008f2034f00d164881 2023-04-06 17:58:57,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/19fe670834af43caa758a2804fb1346c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/19fe670834af43caa758a2804fb1346c 2023-04-06 17:58:57,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/b87bece5b2e64d4ba78b1e0cb25c3113 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/b87bece5b2e64d4ba78b1e0cb25c3113 2023-04-06 17:58:57,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e647d5cddac141688e23a2e96183121d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/e647d5cddac141688e23a2e96183121d 2023-04-06 17:58:57,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0853b9e7d69a425fabb23397c9afd15d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/0853b9e7d69a425fabb23397c9afd15d 2023-04-06 17:58:57,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90ee65981ea848319d69122465096483 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90ee65981ea848319d69122465096483 2023-04-06 17:58:57,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/11b180ed85154e76888aa25821989cc3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/11b180ed85154e76888aa25821989cc3 2023-04-06 17:58:57,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/802af9dc4247476c8c957d602d33234b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/802af9dc4247476c8c957d602d33234b 2023-04-06 17:58:57,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c33ca04fc4db42359cb4b2bf8f747db1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/0ee406bb9ae44bcc971ec464cbf30da4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a7fe282016d4d3f91a741d9f475aaf2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c7e544ea823a4025a7d9f35fa8e5fcf9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/a43572275b37467bb7fe3e1cb8726b7c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/815046ac35c045de89f0c4a3e99edcfc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/474ccbfe69414315a279cb52750e9946, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/cd2eefa1650d4500a598a7641b28f496, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b599f7187f1b47fe86b84918e47d087c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/2e4f3ef60a8f4617a70691ad1fb6b3c5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4abbfd26fb21430fb45f235c423b0a90, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/efae3397841e422fbaa92a56af72c820, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b467a484ba9f4790bd30b01436d83949, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/314a4a5edb4e414781fc5e606acdc697, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b439f858eca345b9bca39894a283dbbb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/d9a0ff0756674ca4a440e73dfeec12ca, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/44593d3454e3401ab1ac2bd9b0974ddb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/8c3e83144a33433d9a8c72f5af3db00e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/057a332a3e974dd19c07ed29dcab482b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/ce49560a092740a6b3b1359abbdd6379, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b98ebd2297e64ab0ad3766972df8b3de, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/7749191c3785437c9f75b66451daa636, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a700c91d3694ed389b5fa23e0c3e2fb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/f47534ce641e4256add6131fa1ce3650, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/db303f4027084104a14ccecd9f572f0c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/aa63c7ad71cd44bb92f3639a8d28a246, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/94ec6d3bd71746db8b969619becd237e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/6ada11537fa341f996957315f313e51e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c35d7c9234eb43a7889468245e641d0a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/e1b146dd0ef74164a7beb64a70037782, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4084934601d643ec984b9a747ccfc1b7] to archive 2023-04-06 17:58:57,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:57,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c33ca04fc4db42359cb4b2bf8f747db1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c33ca04fc4db42359cb4b2bf8f747db1 2023-04-06 17:58:57,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/0ee406bb9ae44bcc971ec464cbf30da4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/0ee406bb9ae44bcc971ec464cbf30da4 2023-04-06 17:58:57,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a7fe282016d4d3f91a741d9f475aaf2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a7fe282016d4d3f91a741d9f475aaf2 2023-04-06 17:58:57,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c7e544ea823a4025a7d9f35fa8e5fcf9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c7e544ea823a4025a7d9f35fa8e5fcf9 2023-04-06 17:58:57,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/a43572275b37467bb7fe3e1cb8726b7c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/a43572275b37467bb7fe3e1cb8726b7c 2023-04-06 17:58:57,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/815046ac35c045de89f0c4a3e99edcfc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/815046ac35c045de89f0c4a3e99edcfc 2023-04-06 17:58:57,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/474ccbfe69414315a279cb52750e9946 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/474ccbfe69414315a279cb52750e9946 2023-04-06 17:58:57,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/cd2eefa1650d4500a598a7641b28f496 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/cd2eefa1650d4500a598a7641b28f496 2023-04-06 17:58:57,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b599f7187f1b47fe86b84918e47d087c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b599f7187f1b47fe86b84918e47d087c 2023-04-06 17:58:57,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/2e4f3ef60a8f4617a70691ad1fb6b3c5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/2e4f3ef60a8f4617a70691ad1fb6b3c5 2023-04-06 17:58:57,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4abbfd26fb21430fb45f235c423b0a90 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4abbfd26fb21430fb45f235c423b0a90 2023-04-06 17:58:57,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/efae3397841e422fbaa92a56af72c820 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/efae3397841e422fbaa92a56af72c820 2023-04-06 17:58:57,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b467a484ba9f4790bd30b01436d83949 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b467a484ba9f4790bd30b01436d83949 2023-04-06 17:58:57,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/314a4a5edb4e414781fc5e606acdc697 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/314a4a5edb4e414781fc5e606acdc697 2023-04-06 17:58:57,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b439f858eca345b9bca39894a283dbbb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b439f858eca345b9bca39894a283dbbb 2023-04-06 17:58:57,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/d9a0ff0756674ca4a440e73dfeec12ca to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/d9a0ff0756674ca4a440e73dfeec12ca 2023-04-06 17:58:57,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/44593d3454e3401ab1ac2bd9b0974ddb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/44593d3454e3401ab1ac2bd9b0974ddb 2023-04-06 17:58:57,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/8c3e83144a33433d9a8c72f5af3db00e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/8c3e83144a33433d9a8c72f5af3db00e 2023-04-06 17:58:57,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/057a332a3e974dd19c07ed29dcab482b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/057a332a3e974dd19c07ed29dcab482b 2023-04-06 17:58:57,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/ce49560a092740a6b3b1359abbdd6379 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/ce49560a092740a6b3b1359abbdd6379 2023-04-06 17:58:57,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b98ebd2297e64ab0ad3766972df8b3de to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/b98ebd2297e64ab0ad3766972df8b3de 2023-04-06 17:58:57,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/7749191c3785437c9f75b66451daa636 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/7749191c3785437c9f75b66451daa636 2023-04-06 17:58:57,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a700c91d3694ed389b5fa23e0c3e2fb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/3a700c91d3694ed389b5fa23e0c3e2fb 2023-04-06 17:58:57,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/f47534ce641e4256add6131fa1ce3650 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/f47534ce641e4256add6131fa1ce3650 2023-04-06 17:58:57,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/db303f4027084104a14ccecd9f572f0c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/db303f4027084104a14ccecd9f572f0c 2023-04-06 17:58:57,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/aa63c7ad71cd44bb92f3639a8d28a246 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/aa63c7ad71cd44bb92f3639a8d28a246 2023-04-06 17:58:57,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/94ec6d3bd71746db8b969619becd237e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/94ec6d3bd71746db8b969619becd237e 2023-04-06 17:58:57,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/6ada11537fa341f996957315f313e51e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/6ada11537fa341f996957315f313e51e 2023-04-06 17:58:57,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c35d7c9234eb43a7889468245e641d0a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/c35d7c9234eb43a7889468245e641d0a 2023-04-06 17:58:57,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/e1b146dd0ef74164a7beb64a70037782 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/e1b146dd0ef74164a7beb64a70037782 2023-04-06 17:58:57,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4084934601d643ec984b9a747ccfc1b7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/4084934601d643ec984b9a747ccfc1b7 2023-04-06 17:58:57,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b22f1ff24d2440a7b4c0ff34c9ca1318, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/4de01fd894ce49ce8294306bcf337bd8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6c45d7c799024603936b821e2c851ecc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/620710eff8c84975b469e826d090eade, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/efed95fd3eed449896bc12456a52aaa5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/8b0dcae0f7474c1ca7af6fc35de33d2b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ad97722813d04c7994f9f9f247c341d1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ac4c3108855748b4a9c5b7fd5aa2c052, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/01469994e3a7411e8587ff721fb6e668, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/69cb2560fa84419e8445be2aad3fc6c5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e268ac4e0db45218dc92a6e63e418aa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/44bf4eec7927414aad42ddce646f79e8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e7c9a7d83d0b4f5b837abb6cd16d8929, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6fa09f9178244960904845e354fedcdd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7a8402b9a1ad4f4182c621a2ff3a7958, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e4d727e763994b81be840826a584463a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5e6d10ac9de144ad9dea4df78fe9ded6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/dca42489751a46f0bad56e11c82b3a8c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/a949687446f045db9d546bf2902bc8b1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5d64894ce16549f1bbc72b54db8db36c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/0447964ef8964048b1d256df1a20b562, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/c8ea5108671e48028ca76c0e00835e5d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2cdce89b22ec405ba6bd3162bfba6ce2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/db40a0daabd546b0b93d9115cc349358, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/fec80d83007f4bbc9ce99a14486d58bb, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7dadabd9bb354d4f83949614233546b7, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3a378637adae470c820b8e463bcbf04c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b97d254e1a0e4351b775b37ce1fa9649, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e946dbbb141432da6baf1ae14861aff, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6a9f20fe63fb404185bbbece7ac3a398, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2602c3304bef431fa3cc3c4048ce521d] to archive 2023-04-06 17:58:57,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:58:57,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b22f1ff24d2440a7b4c0ff34c9ca1318 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b22f1ff24d2440a7b4c0ff34c9ca1318 2023-04-06 17:58:57,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/4de01fd894ce49ce8294306bcf337bd8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/4de01fd894ce49ce8294306bcf337bd8 2023-04-06 17:58:57,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6c45d7c799024603936b821e2c851ecc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6c45d7c799024603936b821e2c851ecc 2023-04-06 17:58:57,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/620710eff8c84975b469e826d090eade to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/620710eff8c84975b469e826d090eade 2023-04-06 17:58:57,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/efed95fd3eed449896bc12456a52aaa5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/efed95fd3eed449896bc12456a52aaa5 2023-04-06 17:58:57,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/8b0dcae0f7474c1ca7af6fc35de33d2b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/8b0dcae0f7474c1ca7af6fc35de33d2b 2023-04-06 17:58:57,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ad97722813d04c7994f9f9f247c341d1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ad97722813d04c7994f9f9f247c341d1 2023-04-06 17:58:57,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ac4c3108855748b4a9c5b7fd5aa2c052 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/ac4c3108855748b4a9c5b7fd5aa2c052 2023-04-06 17:58:57,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/01469994e3a7411e8587ff721fb6e668 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/01469994e3a7411e8587ff721fb6e668 2023-04-06 17:58:57,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/69cb2560fa84419e8445be2aad3fc6c5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/69cb2560fa84419e8445be2aad3fc6c5 2023-04-06 17:58:57,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e268ac4e0db45218dc92a6e63e418aa to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e268ac4e0db45218dc92a6e63e418aa 2023-04-06 17:58:57,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/44bf4eec7927414aad42ddce646f79e8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/44bf4eec7927414aad42ddce646f79e8 2023-04-06 17:58:57,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e7c9a7d83d0b4f5b837abb6cd16d8929 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e7c9a7d83d0b4f5b837abb6cd16d8929 2023-04-06 17:58:57,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6fa09f9178244960904845e354fedcdd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6fa09f9178244960904845e354fedcdd 2023-04-06 17:58:57,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7a8402b9a1ad4f4182c621a2ff3a7958 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7a8402b9a1ad4f4182c621a2ff3a7958 2023-04-06 17:58:57,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e4d727e763994b81be840826a584463a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/e4d727e763994b81be840826a584463a 2023-04-06 17:58:57,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5e6d10ac9de144ad9dea4df78fe9ded6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5e6d10ac9de144ad9dea4df78fe9ded6 2023-04-06 17:58:57,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/dca42489751a46f0bad56e11c82b3a8c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/dca42489751a46f0bad56e11c82b3a8c 2023-04-06 17:58:57,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/a949687446f045db9d546bf2902bc8b1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/a949687446f045db9d546bf2902bc8b1 2023-04-06 17:58:57,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5d64894ce16549f1bbc72b54db8db36c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5d64894ce16549f1bbc72b54db8db36c 2023-04-06 17:58:57,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/0447964ef8964048b1d256df1a20b562 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/0447964ef8964048b1d256df1a20b562 2023-04-06 17:58:57,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/c8ea5108671e48028ca76c0e00835e5d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/c8ea5108671e48028ca76c0e00835e5d 2023-04-06 17:58:57,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2cdce89b22ec405ba6bd3162bfba6ce2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2cdce89b22ec405ba6bd3162bfba6ce2 2023-04-06 17:58:57,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/db40a0daabd546b0b93d9115cc349358 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/db40a0daabd546b0b93d9115cc349358 2023-04-06 17:58:57,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/fec80d83007f4bbc9ce99a14486d58bb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/fec80d83007f4bbc9ce99a14486d58bb 2023-04-06 17:58:57,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7dadabd9bb354d4f83949614233546b7 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/7dadabd9bb354d4f83949614233546b7 2023-04-06 17:58:57,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3a378637adae470c820b8e463bcbf04c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3a378637adae470c820b8e463bcbf04c 2023-04-06 17:58:57,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b97d254e1a0e4351b775b37ce1fa9649 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/b97d254e1a0e4351b775b37ce1fa9649 2023-04-06 17:58:57,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e946dbbb141432da6baf1ae14861aff to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/3e946dbbb141432da6baf1ae14861aff 2023-04-06 17:58:57,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6a9f20fe63fb404185bbbece7ac3a398 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/6a9f20fe63fb404185bbbece7ac3a398 2023-04-06 17:58:57,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2602c3304bef431fa3cc3c4048ce521d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/2602c3304bef431fa3cc3c4048ce521d 2023-04-06 17:58:57,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/recovered.edits/462.seqid, newMaxSeqId=462, maxSeqId=1 2023-04-06 17:58:57,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d. 2023-04-06 17:58:57,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ec62d27e82d7188e5d149482edb0608d: 2023-04-06 17:58:57,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:57,121 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ec62d27e82d7188e5d149482edb0608d, regionState=CLOSED 2023-04-06 17:58:57,121 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803937121"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803937121"}]},"ts":"1680803937121"} 2023-04-06 17:58:57,123 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-06 17:58:57,124 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure ec62d27e82d7188e5d149482edb0608d, server=jenkins-hbase20.apache.org,38773,1680803776439 in 1.5480 sec 2023-04-06 17:58:57,125 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-06 17:58:57,125 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec62d27e82d7188e5d149482edb0608d, UNASSIGN in 1.5520 sec 2023-04-06 17:58:57,125 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803937125"}]},"ts":"1680803937125"} 2023-04-06 17:58:57,126 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 17:58:57,127 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 17:58:57,129 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5720 sec 2023-04-06 17:58:57,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 17:58:57,668 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-06 17:58:57,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 17:58:57,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:57,671 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:57,672 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:57,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-06 17:58:57,678 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:57,682 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/recovered.edits] 2023-04-06 17:58:57,687 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/621316f148b94801b750b9d68cab4d42 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/621316f148b94801b750b9d68cab4d42 2023-04-06 17:58:57,689 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90381db0f7364206b5592ab0a8146a5e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/A/90381db0f7364206b5592ab0a8146a5e 2023-04-06 17:58:57,693 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/73d67aa9c8e347d89287cf87e17f7354 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/73d67aa9c8e347d89287cf87e17f7354 2023-04-06 17:58:57,696 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/76d2ab9c222d4e42959ef87af778e2e6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/B/76d2ab9c222d4e42959ef87af778e2e6 2023-04-06 17:58:57,700 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5ee5cbb4cfd342f58a96107e5eb88d40 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/5ee5cbb4cfd342f58a96107e5eb88d40 2023-04-06 17:58:57,703 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/abfc2c668fc443968a8bf471ba48edef to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/C/abfc2c668fc443968a8bf471ba48edef 2023-04-06 17:58:57,708 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/recovered.edits/462.seqid to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d/recovered.edits/462.seqid 2023-04-06 17:58:57,710 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/ec62d27e82d7188e5d149482edb0608d 2023-04-06 17:58:57,710 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:58:57,714 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:57,717 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 17:58:57,719 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 17:58:57,720 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:57,720 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 17:58:57,721 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680803937720"}]},"ts":"9223372036854775807"} 2023-04-06 17:58:57,723 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 17:58:57,723 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ec62d27e82d7188e5d149482edb0608d, NAME => 'TestAcidGuarantees,,1680803906917.ec62d27e82d7188e5d149482edb0608d.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 17:58:57,723 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 17:58:57,723 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680803937723"}]},"ts":"9223372036854775807"} 2023-04-06 17:58:57,726 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 17:58:57,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:57,730 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-04-06 17:58:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-06 17:58:57,774 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-06 17:58:57,783 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=543 (was 512) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 417), ProcessCount=167 (was 167), AvailableMemoryMB=3816 (was 3751) - AvailableMemoryMB LEAK? - 2023-04-06 17:58:57,793 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=543, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=167, AvailableMemoryMB=3816 2023-04-06 17:58:57,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:58:57,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:58:57,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 17:58:57,797 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 17:58:57,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-06 17:58:57,798 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 17:58:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 17:58:57,799 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:57,799 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 empty. 2023-04-06 17:58:57,800 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:57,800 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:58:57,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 17:58:58,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 17:58:58,208 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:58:58,210 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7893bef985668400d398f4c26aaccd85, NAME => 'TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp 2023-04-06 17:58:58,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 17:58:58,615 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:58:58,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7893bef985668400d398f4c26aaccd85, disabling compactions & flushes 2023-04-06 17:58:58,616 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. after waiting 0 ms 2023-04-06 17:58:58,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,616 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:58:58,622 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 17:58:58,624 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803938624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803938624"}]},"ts":"1680803938624"} 2023-04-06 17:58:58,626 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 17:58:58,627 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 17:58:58,628 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803938627"}]},"ts":"1680803938627"} 2023-04-06 17:58:58,629 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 17:58:58,633 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, ASSIGN}] 2023-04-06 17:58:58,636 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, ASSIGN 2023-04-06 17:58:58,636 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=false 2023-04-06 17:58:58,789 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:58,789 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803938789"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803938789"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803938789"}]},"ts":"1680803938789"} 2023-04-06 17:58:58,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:58:58,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 17:58:58,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7893bef985668400d398f4c26aaccd85, NAME => 'TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:58:58,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:58:58,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,951 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,953 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A 2023-04-06 17:58:58,953 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A 2023-04-06 17:58:58,953 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:58:58,954 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7893bef985668400d398f4c26aaccd85 columnFamilyName A 2023-04-06 17:58:58,954 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(310): Store=7893bef985668400d398f4c26aaccd85/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:58:58,954 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,956 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B 2023-04-06 17:58:58,956 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B 2023-04-06 17:58:58,956 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:58:58,956 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7893bef985668400d398f4c26aaccd85 columnFamilyName B 2023-04-06 17:58:58,956 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(310): Store=7893bef985668400d398f4c26aaccd85/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:58:58,957 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,957 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C 2023-04-06 17:58:58,958 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C 2023-04-06 17:58:58,958 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:58:58,958 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7893bef985668400d398f4c26aaccd85 columnFamilyName C 2023-04-06 17:58:58,958 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(310): Store=7893bef985668400d398f4c26aaccd85/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:58:58,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:58:58,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 7893bef985668400d398f4c26aaccd85 2023-04-06 17:58:58,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 17:58:58,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 7893bef985668400d398f4c26aaccd85; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62798646, jitterRate=-0.06422725319862366}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:58:58,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:58:58,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., pid=56, masterSystemTime=1680803938945 2023-04-06 17:58:58,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:58:58,965 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:58:58,965 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803938965"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803938965"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803938965"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803938965"}]},"ts":"1680803938965"} 2023-04-06 17:58:58,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-06 17:58:58,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 in 174 msec 2023-04-06 17:58:58,968 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-06 17:58:58,968 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, ASSIGN in 334 msec 2023-04-06 17:58:58,968 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 17:58:58,968 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803938968"}]},"ts":"1680803938968"} 2023-04-06 17:58:58,968 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 17:58:58,974 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 17:58:58,975 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1790 sec 2023-04-06 17:58:59,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 17:58:59,907 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-06 17:58:59,908 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x25d2db45 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:58:59,917 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64d8b1e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:58:59,919 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:58:59,920 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:58:59,922 DEBUG [Listener at localhost.localdomain/40169] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 17:58:59,925 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 17:58:59,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 17:58:59,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 17:58:59,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:00,346 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 17:59:00,349 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 17:59:00,349 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 17:59:00,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 17:59:00,359 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, REOPEN/MOVE}] 2023-04-06 17:59:00,360 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, REOPEN/MOVE 2023-04-06 17:59:00,361 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,361 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803940361"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803940361"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803940361"}]},"ts":"1680803940361"} 2023-04-06 17:59:00,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:59:00,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 7893bef985668400d398f4c26aaccd85, disabling compactions & flushes 2023-04-06 17:59:00,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. after waiting 0 ms 2023-04-06 17:59:00,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 17:59:00,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:00,525 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 7893bef985668400d398f4c26aaccd85 to self. 2023-04-06 17:59:00,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,529 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=CLOSED 2023-04-06 17:59:00,529 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803940529"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803940529"}]},"ts":"1680803940529"} 2023-04-06 17:59:00,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-06 17:59:00,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 in 168 msec 2023-04-06 17:59:00,534 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,38773,1680803776439; forceNewPlan=false, retain=true 2023-04-06 17:59:00,684 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,685 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803940684"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803940684"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803940684"}]},"ts":"1680803940684"} 2023-04-06 17:59:00,686 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:59:00,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7893bef985668400d398f4c26aaccd85, NAME => 'TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.', STARTKEY => '', ENDKEY => ''} 2023-04-06 17:59:00,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 17:59:00,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,844 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,846 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A 2023-04-06 17:59:00,846 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A 2023-04-06 17:59:00,846 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:59:00,847 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7893bef985668400d398f4c26aaccd85 columnFamilyName A 2023-04-06 17:59:00,849 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(310): Store=7893bef985668400d398f4c26aaccd85/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:59:00,850 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,851 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B 2023-04-06 17:59:00,851 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B 2023-04-06 17:59:00,852 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:59:00,852 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7893bef985668400d398f4c26aaccd85 columnFamilyName B 2023-04-06 17:59:00,853 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(310): Store=7893bef985668400d398f4c26aaccd85/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:59:00,853 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,854 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C 2023-04-06 17:59:00,854 DEBUG [StoreOpener-7893bef985668400d398f4c26aaccd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C 2023-04-06 17:59:00,855 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 17:59:00,855 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7893bef985668400d398f4c26aaccd85 columnFamilyName C 2023-04-06 17:59:00,855 INFO [StoreOpener-7893bef985668400d398f4c26aaccd85-1] regionserver.HStore(310): Store=7893bef985668400d398f4c26aaccd85/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 17:59:00,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 17:59:00,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 7893bef985668400d398f4c26aaccd85; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65176514, jitterRate=-0.02879425883293152}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 17:59:00,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:00,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., pid=61, masterSystemTime=1680803940837 2023-04-06 17:59:00,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,866 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,867 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803940866"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680803940866"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680803940866"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680803940866"}]},"ts":"1680803940866"} 2023-04-06 17:59:00,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-06 17:59:00,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 in 182 msec 2023-04-06 17:59:00,872 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-06 17:59:00,872 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, REOPEN/MOVE in 511 msec 2023-04-06 17:59:00,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-06 17:59:00,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 519 msec 2023-04-06 17:59:00,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 947 msec 2023-04-06 17:59:00,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-06 17:59:00,879 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x2c2ce5cd to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,884 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f970bfe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,884 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x082ccebe to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,889 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43e540fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,889 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x5768f049 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,894 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@395f9d24, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,894 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x67b5579f to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,899 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@774f1998, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,900 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x451cdd43 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,905 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@85873d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,905 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x768816e9 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,915 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d853d1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,915 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x27acfde4 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,921 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@112588a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,922 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x00fcbd84 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,931 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@407f33a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,932 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x18884189 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,938 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7dfddf81, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,938 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(139): Connect 0x2ef34de3 to 127.0.0.1:49384 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 17:59:00,948 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74adc5bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 17:59:00,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:00,957 DEBUG [hconnection-0x3b21f98a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,960 DEBUG [hconnection-0x62ecfdd3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,964 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,964 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,964 DEBUG [hconnection-0x72f8b65a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,964 DEBUG [hconnection-0x455451f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,965 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,966 DEBUG [hconnection-0x4a3a5fb8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,966 DEBUG [hconnection-0x7e5401aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,966 DEBUG [hconnection-0x4527562e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,966 DEBUG [hconnection-0x1b3168db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,966 DEBUG [hconnection-0x2aff626f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:00,973 DEBUG [hconnection-0x2db45121-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 17:59:00,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:00,975 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,975 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,975 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,975 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,975 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:00,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:00,976 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,975 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57334, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,976 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:00,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:00,984 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 17:59:00,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:00,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:00,985 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,985 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:00,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:00,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,986 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:00,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:59:00,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:00,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:00,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:00,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:00,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:00,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:00,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:00,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:00,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:00,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:00,991 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,991 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:00,991 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:00,991 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:00,991 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:00,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:00,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:00,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:00,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:00,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:00,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:00,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c89f480[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:00,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,004 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,004 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,005 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,005 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,008 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,008 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,008 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,009 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,009 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,009 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,010 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803941004, max timestamp=1680803941009Num uniques 859; 2023-04-06 17:59:01,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:01,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803941004, max timestamp=1680803941009Num uniques 859; 2023-04-06 17:59:01,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,013 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803941004, max timestamp=1680803941009Num uniques 859; 2023-04-06 17:59:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:01,016 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,016 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,016 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,016 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,022 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ddf0dc700cbf46a98c8c6731bb005015 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ddf0dc700cbf46a98c8c6731bb005015 2023-04-06 17:59:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804001026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804001027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804001028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804001028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804001029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,034 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/ea90f483d33f4ffd9315c92a4e062174 2023-04-06 17:59:01,116 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,119 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,120 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:01,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,121 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,121 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,121 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:01,121 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,121 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:01,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:01,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,122 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:01,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:01,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a62e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:01,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,124 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,125 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,127 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,127 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,128 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,128 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,128 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,129 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,130 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:01,130 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,131 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,131 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804001133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804001133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804001133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804001133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804001134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,231 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,234 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,235 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:01,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,236 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,237 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,237 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,237 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:01,237 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:01,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,237 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,238 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,238 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:01,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:01,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c643c2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:01,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,239 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,240 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,240 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,242 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,242 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,242 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,246 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,246 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,246 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,247 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804001336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804001336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804001336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804001337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804001337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,346 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,349 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,350 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:01,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,351 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:01,352 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:01,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:01,352 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,353 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:01,353 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:01,353 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,353 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,354 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:01,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f741a78[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:01,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:01,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,356 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,356 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,356 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,357 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,358 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,358 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,358 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,359 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,360 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,361 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,363 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,363 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,363 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,363 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/816db5813a8f4dfaa4c7699a5ad70b95 2023-04-06 17:59:01,463 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,466 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,468 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,469 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:01,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,470 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:01,471 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:01,481 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:01,482 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:01,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cbadce2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:01,482 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,483 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,483 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,483 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,484 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,484 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,486 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,486 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,487 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,487 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,487 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,488 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,489 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:01,490 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,491 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,491 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,491 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,591 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,594 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,594 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,595 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,595 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:01,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,596 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,596 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,596 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:01,596 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:01,597 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51af518d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:01,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:01,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,599 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,599 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,600 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,601 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,601 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,606 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:01,606 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804001638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804001638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804001638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804001641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:01,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804001641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,706 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,708 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,710 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,710 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:01,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,710 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,711 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,711 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,711 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:01,711 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:01,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:01,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:01,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:01,711 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,711 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,712 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:01,713 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:01,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b9dd029[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:01,713 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,713 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,713 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,713 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,714 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,714 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,716 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,716 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,716 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,717 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,717 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,719 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,719 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,720 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,721 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:01,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,722 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,722 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,722 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,822 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,824 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,824 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,825 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,825 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,825 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,825 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,826 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,826 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,826 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,826 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,826 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,827 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,827 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,827 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,827 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:01,827 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,827 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:01,827 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:01,827 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,827 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:01,827 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:01,827 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,828 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,828 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,828 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:01,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:01,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6465baf8[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:01,829 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,830 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:01,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,830 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:01,830 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:01,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:01,832 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:01,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:01,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:01,832 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:01,832 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:01,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:01,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,833 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:01,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,834 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,835 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:01,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:01,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,836 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:01,836 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:01,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:01,837 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:01,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,837 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:01,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:01,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/2a9c3806ba5545bb93b94c1de595c8f8 2023-04-06 17:59:01,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/ea90f483d33f4ffd9315c92a4e062174 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ea90f483d33f4ffd9315c92a4e062174 2023-04-06 17:59:01,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ea90f483d33f4ffd9315c92a4e062174, entries=100, sequenceid=15, filesize=18.4 K 2023-04-06 17:59:01,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/816db5813a8f4dfaa4c7699a5ad70b95 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/816db5813a8f4dfaa4c7699a5ad70b95 2023-04-06 17:59:01,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/816db5813a8f4dfaa4c7699a5ad70b95, entries=100, sequenceid=15, filesize=9.3 K 2023-04-06 17:59:01,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/2a9c3806ba5545bb93b94c1de595c8f8 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2a9c3806ba5545bb93b94c1de595c8f8 2023-04-06 17:59:01,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2a9c3806ba5545bb93b94c1de595c8f8, entries=100, sequenceid=15, filesize=9.3 K 2023-04-06 17:59:01,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7893bef985668400d398f4c26aaccd85 in 876ms, sequenceid=15, compaction requested=false 2023-04-06 17:59:01,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:01,937 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:01,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:01,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:01,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:01,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:01,940 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:01,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:01,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:01,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,940 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,941 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:01,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:01,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:01,941 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:01,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:01,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:01,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:01,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:01,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:01,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:01,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:01,944 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:01,944 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:01,944 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:01,945 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-06 17:59:01,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:01,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:01,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:01,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:01,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:01,945 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:02,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:02,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:02,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804002147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804002147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804002148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804002148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804002149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804002249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804002250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804002250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804002251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,355 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b5b2fb4a835a4aeb8b8ed59623a49c45 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b5b2fb4a835a4aeb8b8ed59623a49c45 2023-04-06 17:59:02,359 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/3da143f0255f48d3a2c369d2cfaaaea2 2023-04-06 17:59:02,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804002453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804002453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804002453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804002454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,641 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 17:59:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804002754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804002755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804002755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:02,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804002756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,778 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/f692f75e9b6d48adaf4f75d7c95ab714 2023-04-06 17:59:02,790 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/97c8b30e8be844d1bdd99ffd3e89ec88 2023-04-06 17:59:02,793 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/3da143f0255f48d3a2c369d2cfaaaea2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3da143f0255f48d3a2c369d2cfaaaea2 2023-04-06 17:59:02,796 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3da143f0255f48d3a2c369d2cfaaaea2, entries=150, sequenceid=51, filesize=25.1 K 2023-04-06 17:59:02,797 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/f692f75e9b6d48adaf4f75d7c95ab714 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f692f75e9b6d48adaf4f75d7c95ab714 2023-04-06 17:59:02,800 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f692f75e9b6d48adaf4f75d7c95ab714, entries=150, sequenceid=51, filesize=11.6 K 2023-04-06 17:59:02,801 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/97c8b30e8be844d1bdd99ffd3e89ec88 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/97c8b30e8be844d1bdd99ffd3e89ec88 2023-04-06 17:59:02,803 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/97c8b30e8be844d1bdd99ffd3e89ec88, entries=150, sequenceid=51, filesize=11.6 K 2023-04-06 17:59:02,804 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 7893bef985668400d398f4c26aaccd85 in 860ms, sequenceid=51, compaction requested=false 2023-04-06 17:59:02,804 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:02,804 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:02,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:02,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:02,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:02,804 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:02,805 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:02,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:02,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,805 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:02,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:02,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef2375a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:02,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:02,807 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,808 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:02,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:02,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:02,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:02,808 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:02,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:02,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:02,809 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:02,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:02,810 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:02,810 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:02,810 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:02,810 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:02,810 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:02,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:02,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:02,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:02,812 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:02,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:02,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:02,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:02,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:02,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:02,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:02,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:02,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,815 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:02,815 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,815 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:02,815 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:02,915 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:02,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:02,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:02,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:02,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:02,917 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:02,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:02,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:02,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:02,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:02,918 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:02,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:02,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:02,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:02,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:02,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:02,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:02,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:02,919 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:02,919 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:02,919 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:02,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:02,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:02,920 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:02,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:03,155 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:03,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803943188, max timestamp=1680803943195Num uniques 859; 2023-04-06 17:59:03,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803943188, max timestamp=1680803943195Num uniques 859; 2023-04-06 17:59:03,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803943188, max timestamp=1680803943195Num uniques 859; 2023-04-06 17:59:03,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:03,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:03,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:03,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804003237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:03,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804003257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:03,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804003257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:03,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804003259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:03,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804003261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:03,329 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c2ee244e28324aae929b9de563089014 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c2ee244e28324aae929b9de563089014 2023-04-06 17:59:03,333 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/43c884a30b594249ad433f806121f8d6 2023-04-06 17:59:03,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804003340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:03,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804003544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:03,740 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/28c35d570da14a39ae607b6ed409a809 2023-04-06 17:59:03,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804003847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,148 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/810e3c80baf24e67af99b67d95a82cb3 2023-04-06 17:59:04,151 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/43c884a30b594249ad433f806121f8d6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/43c884a30b594249ad433f806121f8d6 2023-04-06 17:59:04,153 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/43c884a30b594249ad433f806121f8d6, entries=150, sequenceid=62, filesize=25.1 K 2023-04-06 17:59:04,154 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/28c35d570da14a39ae607b6ed409a809 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/28c35d570da14a39ae607b6ed409a809 2023-04-06 17:59:04,157 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/28c35d570da14a39ae607b6ed409a809, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 17:59:04,157 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/810e3c80baf24e67af99b67d95a82cb3 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/810e3c80baf24e67af99b67d95a82cb3 2023-04-06 17:59:04,160 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/810e3c80baf24e67af99b67d95a82cb3, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 17:59:04,160 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7893bef985668400d398f4c26aaccd85 in 1240ms, sequenceid=62, compaction requested=true 2023-04-06 17:59:04,160 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:04,160 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,160 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,160 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,160 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,161 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,161 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,161 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,163 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa2830[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,163 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,163 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,164 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:04,164 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:04,164 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:04,164 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,164 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:04,164 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:04,164 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,164 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,164 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:04,166 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:04,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:04,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:04,166 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:04,167 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:04,167 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,167 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,167 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,168 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,169 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,170 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:04,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:04,170 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,171 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:04,171 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:04,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:04,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:04,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-06 17:59:04,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:04,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:04,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:04,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:04,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:04,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:04,271 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:04,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:04,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804004270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:04,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804004269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:04,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:04,274 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:04,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:04,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:04,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,274 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,275 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804004272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804004273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,277 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,277 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:04,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:04,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:04,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:04,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:04,278 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,278 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:04,278 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:04,278 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406e295c7d6eab242e4b6e6a6e6a58fcf6c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e295c7d6eab242e4b6e6a6e6a58fcf6c 2023-04-06 17:59:04,278 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:04,278 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,279 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,280 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,280 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,280 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,280 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b3ac3e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,281 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,281 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,282 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,282 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,282 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:04,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:04,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:04,282 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:04,283 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/5ef77f34540b45aeb6d939939bec97bd 2023-04-06 17:59:04,284 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:04,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,284 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:04,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:04,286 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:04,286 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:04,286 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:04,287 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:04,287 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:04,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,296 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,296 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,297 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,299 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:04,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:04,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,299 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,299 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:04,300 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:04,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804004354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804004374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804004374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804004376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804004376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,400 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:04,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:04,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:04,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:04,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:04,402 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:04,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:04,402 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:04,403 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,403 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,403 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,403 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,403 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,403 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:04,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:04,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:04,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:04,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,404 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:04,405 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,405 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:04,405 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:04,405 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,406 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,406 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c86eaa[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,407 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,407 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,408 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:04,408 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:04,408 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,408 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:04,408 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:04,408 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,408 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,409 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,409 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:04,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:04,411 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:04,411 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:04,411 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:04,411 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:04,411 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:04,412 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,412 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,412 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,412 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,413 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:04,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:04,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,414 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,414 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,415 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,415 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:04,415 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:04,415 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:04,515 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:04,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:04,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:04,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:04,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:04,517 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:04,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:04,517 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:04,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,518 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,518 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:04,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:04,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:04,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:04,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:04,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,519 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:04,520 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,520 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:04,520 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:04,520 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,521 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,521 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263fd22e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,522 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,523 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:04,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:04,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,523 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:04,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:04,523 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:04,523 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,524 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:04,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:04,525 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:04,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:04,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:04,525 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:04,525 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:04,526 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,526 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,526 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:04,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,529 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,529 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:04,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,529 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:04,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804004575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804004576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804004577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804004577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,629 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:04,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:04,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:04,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:04,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:04,632 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:04,632 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,632 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:04,632 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:04,632 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,632 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,633 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,633 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,633 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,633 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:04,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:04,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:04,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:04,634 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:04,634 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,634 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:04,634 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:04,634 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:04,634 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,634 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:04,634 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:04,634 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,634 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,635 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,635 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,635 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,635 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,635 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,635 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,636 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,636 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7250c571[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,637 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,638 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:04,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:04,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:04,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,638 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:04,638 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,638 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,639 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:04,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:04,640 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:04,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:04,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:04,640 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:04,640 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:04,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,641 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,642 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,643 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,645 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,645 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,645 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,645 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:04,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:04,645 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,645 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,646 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:04,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,646 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:04,646 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,692 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/5e4d3e564dc449ed949d14f2c882eb59 2023-04-06 17:59:04,746 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:04,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:04,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:04,749 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:04,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:04,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:04,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,749 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,750 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,750 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,750 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,750 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:04,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:04,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:04,751 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:04,751 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,751 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:04,751 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:04,751 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,751 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:04,751 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:04,751 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:04,751 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,751 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,752 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,752 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,753 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,753 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f4f5559[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,754 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:04,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:04,754 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:04,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,754 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:04,755 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:04,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,755 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:04,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:04,757 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:04,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:04,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:04,757 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:04,757 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:04,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,758 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,758 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,759 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,761 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:04,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,761 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,761 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:04,761 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:04,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,861 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:04,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:04,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:04,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:04,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:04,864 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:04,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:04,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:04,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,865 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,865 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:04,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:04,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:04,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:04,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:04,866 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:04,867 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:04,867 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:04,867 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:04,867 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,867 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,868 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,868 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6528d124[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,869 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,870 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:04,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:04,870 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:04,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,870 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:04,871 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:04,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,871 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:04,873 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:04,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:04,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:04,873 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:04,873 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:04,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,873 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,874 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,875 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,876 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:04,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:04,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:04,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:04,876 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,877 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:04,877 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:04,877 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:04,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804004877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804004879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804004879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:04,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804004881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,977 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:04,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:04,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:04,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:04,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:04,980 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:04,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:04,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:04,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,980 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,981 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:04,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:04,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,981 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:04,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:04,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:04,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:04,993 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,993 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:04,993 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:04,993 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:04,993 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:04,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,997 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:04,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:04,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:04,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:04,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d5ec10d[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:04,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,999 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:04,999 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:05,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:05,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:05,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:05,000 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:05,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:05,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:05,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:05,002 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:05,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:05,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:05,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:05,002 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:05,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:05,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:05,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:05,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:05,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:05,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:05,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:05,006 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:05,007 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:05,007 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:05,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:05,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:05,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:05,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:05,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:05,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/85fc2597550a4f5880e951e8c6d0410b 2023-04-06 17:59:05,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/5ef77f34540b45aeb6d939939bec97bd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5ef77f34540b45aeb6d939939bec97bd 2023-04-06 17:59:05,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5ef77f34540b45aeb6d939939bec97bd, entries=250, sequenceid=101, filesize=38.7 K 2023-04-06 17:59:05,107 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:05,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/5e4d3e564dc449ed949d14f2c882eb59 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/5e4d3e564dc449ed949d14f2c882eb59 2023-04-06 17:59:05,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:05,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:05,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:05,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:05,109 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:05,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:05,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:05,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:05,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/5e4d3e564dc449ed949d14f2c882eb59, entries=150, sequenceid=101, filesize=11.6 K 2023-04-06 17:59:05,111 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:05,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/85fc2597550a4f5880e951e8c6d0410b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/85fc2597550a4f5880e951e8c6d0410b 2023-04-06 17:59:05,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:05,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,111 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:05,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:05,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:05,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:05,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:05,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:05,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:05,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:05,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/85fc2597550a4f5880e951e8c6d0410b, entries=150, sequenceid=101, filesize=11.6 K 2023-04-06 17:59:05,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:05,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,114 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:05,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=33.54 KB/34350 for 7893bef985668400d398f4c26aaccd85 in 849ms, sequenceid=101, compaction requested=true 2023-04-06 17:59:05,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,114 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:05,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:05,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:05,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:05,114 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:05,115 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109920 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:05,115 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:05,115 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:05,116 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ea90f483d33f4ffd9315c92a4e062174, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3da143f0255f48d3a2c369d2cfaaaea2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/43c884a30b594249ad433f806121f8d6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5ef77f34540b45aeb6d939939bec97bd] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=107.3 K 2023-04-06 17:59:05,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:05,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4951f09a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:05,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:05,116 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:05,116 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,116 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:05,116 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:05,116 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/816db5813a8f4dfaa4c7699a5ad70b95, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f692f75e9b6d48adaf4f75d7c95ab714, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/28c35d570da14a39ae607b6ed409a809, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/5e4d3e564dc449ed949d14f2c882eb59] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=44.1 K 2023-04-06 17:59:05,116 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ea90f483d33f4ffd9315c92a4e062174, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680803940987 2023-04-06 17:59:05,116 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 816db5813a8f4dfaa4c7699a5ad70b95, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680803940987 2023-04-06 17:59:05,117 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3da143f0255f48d3a2c369d2cfaaaea2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680803941026 2023-04-06 17:59:05,117 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,117 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f692f75e9b6d48adaf4f75d7c95ab714, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680803941026 2023-04-06 17:59:05,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,117 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:05,117 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 43c884a30b594249ad433f806121f8d6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680803942145 2023-04-06 17:59:05,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:05,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:05,117 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 28c35d570da14a39ae607b6ed409a809, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680803942145 2023-04-06 17:59:05,117 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:05,117 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 5ef77f34540b45aeb6d939939bec97bd, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680803943226 2023-04-06 17:59:05,118 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5e4d3e564dc449ed949d14f2c882eb59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680803944264 2023-04-06 17:59:05,118 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:05,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:05,118 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:05,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:05,121 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:05,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:05,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:05,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:05,121 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:05,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:05,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:05,123 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,125 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#509 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,128 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,131 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#508 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:05,136 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/a0e9fda13e6e411d8a3e32706dd901ba as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/a0e9fda13e6e411d8a3e32706dd901ba 2023-04-06 17:59:05,139 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,139 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,139 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:05,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:05,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:05,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:05,140 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:05,139 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:05,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:05,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:05,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:05,139 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:05,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,140 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:05,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:05,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:05,141 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into a0e9fda13e6e411d8a3e32706dd901ba(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:05,141 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:05,141 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=12, startTime=1680803945114; duration=0sec 2023-04-06 17:59:05,141 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:05,141 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:05,142 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:05,142 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:05,142 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:05,142 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2a9c3806ba5545bb93b94c1de595c8f8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/97c8b30e8be844d1bdd99ffd3e89ec88, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/810e3c80baf24e67af99b67d95a82cb3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/85fc2597550a4f5880e951e8c6d0410b] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=44.1 K 2023-04-06 17:59:05,142 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2a9c3806ba5545bb93b94c1de595c8f8, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680803940987 2023-04-06 17:59:05,143 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 97c8b30e8be844d1bdd99ffd3e89ec88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680803941026 2023-04-06 17:59:05,143 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 810e3c80baf24e67af99b67d95a82cb3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680803942145 2023-04-06 17:59:05,143 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 85fc2597550a4f5880e951e8c6d0410b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680803944264 2023-04-06 17:59:05,152 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/cb045c6e4174404a99a15141fb3272f2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/cb045c6e4174404a99a15141fb3272f2 2023-04-06 17:59:05,154 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:05,155 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into cb045c6e4174404a99a15141fb3272f2(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:05,155 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:05,155 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=12, startTime=1680803945114; duration=0sec 2023-04-06 17:59:05,155 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:05,240 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:05,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:05,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:05,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:05,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:05,243 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:05,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:05,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:05,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:05,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,243 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,244 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:05,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,244 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:05,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:05,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:05,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:05,245 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:05,245 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:05,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:05,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:05,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803945387, max timestamp=1680803945392Num uniques 859; 2023-04-06 17:59:05,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:05,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803945387, max timestamp=1680803945392Num uniques 859; 2023-04-06 17:59:05,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:05,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680803945387, max timestamp=1680803945392Num uniques 859; 2023-04-06 17:59:05,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 17:59:05,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680803945402, max timestamp=1680803945406Num uniques 1018; 2023-04-06 17:59:05,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-06 17:59:05,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680803945402, max timestamp=1680803945406Num uniques 1018; 2023-04-06 17:59:05,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-06 17:59:05,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680803945402, max timestamp=1680803945406Num uniques 1018; 2023-04-06 17:59:05,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-06 17:59:05,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804005417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804005418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804005418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804005419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804005419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804005521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804005521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804005523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804005523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804005523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,561 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/6e8cf84dbc324b6e9ee0a72c7c5d0dce as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/6e8cf84dbc324b6e9ee0a72c7c5d0dce 2023-04-06 17:59:05,563 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into 6e8cf84dbc324b6e9ee0a72c7c5d0dce(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:05,563 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:05,563 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=12, startTime=1680803945114; duration=0sec 2023-04-06 17:59:05,564 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:05,653 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406297249748d3f4fd597294ddf6a6d2124 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406297249748d3f4fd597294ddf6a6d2124 2023-04-06 17:59:05,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804005724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804005724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804005724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804005725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:05,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:05,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804005725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804006027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804006028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804006028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804006028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804006030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,065 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/940e9aefd79a4bb5adeb2cba506c29d2 2023-04-06 17:59:06,473 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/4433f00aa02e4d2ab94fadc97bcd2089 2023-04-06 17:59:06,480 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/0c73689c3f544ea69db477751d51827a 2023-04-06 17:59:06,484 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/940e9aefd79a4bb5adeb2cba506c29d2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/940e9aefd79a4bb5adeb2cba506c29d2 2023-04-06 17:59:06,486 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/940e9aefd79a4bb5adeb2cba506c29d2, entries=150, sequenceid=111, filesize=25.1 K 2023-04-06 17:59:06,487 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/4433f00aa02e4d2ab94fadc97bcd2089 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/4433f00aa02e4d2ab94fadc97bcd2089 2023-04-06 17:59:06,489 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/4433f00aa02e4d2ab94fadc97bcd2089, entries=150, sequenceid=111, filesize=11.6 K 2023-04-06 17:59:06,490 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/0c73689c3f544ea69db477751d51827a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0c73689c3f544ea69db477751d51827a 2023-04-06 17:59:06,492 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0c73689c3f544ea69db477751d51827a, entries=150, sequenceid=111, filesize=11.6 K 2023-04-06 17:59:06,493 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 7893bef985668400d398f4c26aaccd85 in 1247ms, sequenceid=111, compaction requested=false 2023-04-06 17:59:06,493 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:06,493 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,493 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:06,493 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:06,493 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,493 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:06,493 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:06,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,494 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:06,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cf18174[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:06,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:06,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,496 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:06,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:06,496 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:06,496 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:06,497 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:06,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:06,499 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:06,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:06,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:06,499 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:06,499 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:06,500 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,500 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,500 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,503 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:06,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:06,503 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:06,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,503 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:06,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:06,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=154.31 KB heapSize=435.23 KB 2023-04-06 17:59:06,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:06,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:06,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:06,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:06,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:06,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804006535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804006535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804006536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804006536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804006536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,541 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040691c507f058e845da8c08ba6cb29c108c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040691c507f058e845da8c08ba6cb29c108c 2023-04-06 17:59:06,604 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:06,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:06,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:06,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:06,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:06,607 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:06,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:06,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:06,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,608 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:06,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:06,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:06,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:06,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:06,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:06,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:06,609 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,609 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:06,609 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:06,609 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:06,610 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:06,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:06,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:06,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:06,610 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,610 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,611 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,611 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:06,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:06,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff3d03f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:06,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,612 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:06,612 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:06,612 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:06,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:06,613 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,613 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:06,613 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,613 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:06,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:06,615 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:06,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:06,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:06,615 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:06,615 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:06,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,616 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,618 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:06,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,618 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:06,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:06,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,619 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:06,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,619 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:06,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804006641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804006641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804006641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804006641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804006641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,719 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:06,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:06,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:06,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:06,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:06,722 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:06,722 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,722 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:06,722 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:06,722 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,722 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,723 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:06,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:06,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:06,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:06,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,724 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:06,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:06,725 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,725 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:06,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:06,725 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:06,725 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:06,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:06,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:06,725 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:06,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,726 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,726 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646e3a65[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:06,727 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:06,727 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,728 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:06,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:06,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,728 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:06,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:06,728 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:06,729 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,729 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:06,729 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:06,730 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:06,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:06,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:06,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:06,730 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:06,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,731 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,733 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:06,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,735 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:06,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,735 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:06,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,835 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:06,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:06,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:06,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:06,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:06,838 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:06,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:06,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:06,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:06,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:06,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:06,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:06,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:06,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:06,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:06,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:06,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,841 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:06,841 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:06,841 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:06,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:06,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:06,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:06,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,842 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:06,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:06,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@227dc352[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:06,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:06,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:06,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:06,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:06,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804006842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:06,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804006842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804006844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,845 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:06,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804006845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:06,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:06,847 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:06,847 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:06,847 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:06,847 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:06,847 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:06,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:06,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804006846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,848 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,848 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,848 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,851 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:06,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:06,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:06,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,852 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,852 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:06,852 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:06,946 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/35dce7d7f5e04f45a4dd46262947847c 2023-04-06 17:59:06,952 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:06,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:06,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:06,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:06,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:06,955 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:06,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:06,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:06,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,956 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,956 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:06,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:06,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:06,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:06,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:06,961 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:06,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:06,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,971 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,971 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,971 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:06,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:06,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22070c48[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:06,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,972 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:06,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:06,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:06,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,973 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:06,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:06,973 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,973 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:06,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:06,975 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:06,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:06,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:06,975 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:06,975 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:06,976 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,976 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,976 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,977 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:06,978 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:06,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:06,979 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:06,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:06,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:06,979 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:06,979 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:06,979 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:06,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,079 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,082 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,082 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,084 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:07,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,084 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,084 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,085 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,085 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,085 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:07,085 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,085 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:07,085 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:07,085 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,085 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:07,085 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:07,086 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,086 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,087 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,087 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:07,088 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:07,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1344521f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:07,088 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,088 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:07,089 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,089 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,089 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:07,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:07,091 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:07,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:07,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:07,091 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:07,091 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:07,092 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,092 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,092 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,092 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,093 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,095 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:07,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:07,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:07,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,095 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,096 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:07,096 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:07,096 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,096 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804007146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804007146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804007147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804007147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804007149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,196 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,198 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,198 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,198 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,198 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,199 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:07,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,200 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,200 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,201 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,201 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:07,201 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,201 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:07,201 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:07,201 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,201 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:07,201 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:07,202 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,202 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:07,203 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:07,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ecff00[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:07,203 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,203 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:07,203 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:07,203 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,204 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:07,203 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:07,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,204 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:07,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:07,206 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:07,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:07,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:07,206 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:07,206 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:07,207 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,207 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,207 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,208 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,209 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,210 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,210 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:07,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:07,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:07,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,211 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,211 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:07,211 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:07,211 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,211 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,311 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,314 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,315 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:07,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,316 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,316 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,316 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:07,316 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:07,316 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:07,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,317 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,317 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,317 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:07,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac7cbdc[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:07,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:07,318 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,319 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:07,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:07,319 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:07,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,319 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:07,320 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:07,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,320 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:07,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:07,322 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:07,322 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:07,322 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:07,322 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:07,322 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:07,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,322 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,323 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,323 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,324 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,325 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:07,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:07,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:07,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,325 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,326 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:07,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,326 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:07,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,326 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/0a85122ed58c489299e186067fa69d02 2023-04-06 17:59:07,426 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,428 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,430 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:07,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,431 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,431 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,431 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:07,431 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:07,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,432 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,432 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:07,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b8eb80[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:07,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:07,433 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,434 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:07,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,434 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:07,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:07,434 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:07,435 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,435 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:07,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:07,437 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:07,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:07,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:07,437 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:07,437 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:07,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,437 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,438 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,438 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,439 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,440 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:07,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:07,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,440 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:07,440 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:07,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,440 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,441 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,541 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,543 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,544 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,544 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:07,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,545 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:07,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:07,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:07,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:07,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:07,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,547 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:07,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615d9f22[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:07,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:07,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,548 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,548 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:07,549 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:07,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:07,551 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:07,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:07,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:07,551 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:07,551 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:07,551 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,551 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,552 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,552 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,553 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:07,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:07,554 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:07,554 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:07,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,554 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:07,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804007650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804007651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804007652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804007652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:07,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804007653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,655 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,657 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:07,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,659 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,659 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,659 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:07,659 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:07,660 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:07,661 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:07,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2878f6[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:07,661 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,662 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:07,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:07,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:07,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:07,662 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:07,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:07,664 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:07,664 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:07,664 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:07,664 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:07,664 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:07,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,665 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,666 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:07,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:07,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:07,668 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:07,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,668 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:07,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,668 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,768 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/fc71e4e449be4e928f99345689f94844 2023-04-06 17:59:07,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,771 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,771 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,771 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,771 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,771 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,771 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,772 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/35dce7d7f5e04f45a4dd46262947847c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/35dce7d7f5e04f45a4dd46262947847c 2023-04-06 17:59:07,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,782 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/35dce7d7f5e04f45a4dd46262947847c, entries=150, sequenceid=167, filesize=25.3 K 2023-04-06 17:59:07,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/0a85122ed58c489299e186067fa69d02 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0a85122ed58c489299e186067fa69d02 2023-04-06 17:59:07,796 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,796 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,796 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,796 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,797 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:07,797 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,797 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:07,797 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:07,797 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,797 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:07,797 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:07,798 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,798 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0a85122ed58c489299e186067fa69d02, entries=150, sequenceid=167, filesize=11.8 K 2023-04-06 17:59:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,798 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,798 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/fc71e4e449be4e928f99345689f94844 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/fc71e4e449be4e928f99345689f94844 2023-04-06 17:59:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:07,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c5ca04[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:07,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:07,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,806 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:07,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,806 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:07,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:07,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:07,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/fc71e4e449be4e928f99345689f94844, entries=150, sequenceid=167, filesize=11.8 K 2023-04-06 17:59:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,809 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~435.19 KB/445632, currentSize=40.25 KB/41220 for 7893bef985668400d398f4c26aaccd85 in 1278ms, sequenceid=167, compaction requested=true 2023-04-06 17:59:07,809 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:07,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:07,810 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:07,810 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:07,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,811 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77526 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:07,811 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:07,811 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:07,811 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:07,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:07,811 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:07,811 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,811 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/a0e9fda13e6e411d8a3e32706dd901ba, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/4433f00aa02e4d2ab94fadc97bcd2089, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0a85122ed58c489299e186067fa69d02] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=35.1 K 2023-04-06 17:59:07,811 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/cb045c6e4174404a99a15141fb3272f2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/940e9aefd79a4bb5adeb2cba506c29d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/35dce7d7f5e04f45a4dd46262947847c] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=75.7 K 2023-04-06 17:59:07,811 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:07,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:07,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:07,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:07,811 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:07,812 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a0e9fda13e6e411d8a3e32706dd901ba, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680803944264 2023-04-06 17:59:07,812 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting cb045c6e4174404a99a15141fb3272f2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680803944264 2023-04-06 17:59:07,813 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4433f00aa02e4d2ab94fadc97bcd2089, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680803944269 2023-04-06 17:59:07,813 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 940e9aefd79a4bb5adeb2cba506c29d2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680803944269 2023-04-06 17:59:07,813 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0a85122ed58c489299e186067fa69d02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680803945417 2023-04-06 17:59:07,813 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 35dce7d7f5e04f45a4dd46262947847c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680803945417 2023-04-06 17:59:07,824 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,824 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,824 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,825 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,827 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:07,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:07,838 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:07,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:07,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,838 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:07,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:07,838 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:07,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:07,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:07,843 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:07,855 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#517 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:07,939 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:07,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:07,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:07,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:07,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:07,942 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:07,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:07,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:07,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,942 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,943 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:07,943 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,944 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:07,944 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:07,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:07,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:07,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:07,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:07,946 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:07,946 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:07,946 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:07,946 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:07,968 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c9d9a2bc2ea346dd80173c8cbab687d0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c9d9a2bc2ea346dd80173c8cbab687d0 2023-04-06 17:59:08,291 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/49909c88022b4591a3a836479a40c763 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49909c88022b4591a3a836479a40c763 2023-04-06 17:59:08,294 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into 49909c88022b4591a3a836479a40c763(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:08,294 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:08,294 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=13, startTime=1680803947810; duration=0sec 2023-04-06 17:59:08,294 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:08,294 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:08,295 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:08,295 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:08,295 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:08,295 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/6e8cf84dbc324b6e9ee0a72c7c5d0dce, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0c73689c3f544ea69db477751d51827a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/fc71e4e449be4e928f99345689f94844] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=35.1 K 2023-04-06 17:59:08,295 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6e8cf84dbc324b6e9ee0a72c7c5d0dce, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680803944264 2023-04-06 17:59:08,296 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0c73689c3f544ea69db477751d51827a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680803944269 2023-04-06 17:59:08,296 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting fc71e4e449be4e928f99345689f94844, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680803945417 2023-04-06 17:59:08,302 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#520 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:08,376 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=176, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/7568d31f320a4a84ba3410fc909a57a5 2023-04-06 17:59:08,404 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/9ab7d7a6103c43bd85e36bcf7ca8823a 2023-04-06 17:59:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:08,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:08,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803948667, max timestamp=1680803948671Num uniques 859; 2023-04-06 17:59:08,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:59:08,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803948667, max timestamp=1680803948671Num uniques 859; 2023-04-06 17:59:08,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803948667, max timestamp=1680803948671Num uniques 859; 2023-04-06 17:59:08,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:59:08,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:59:08,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804008683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804008684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804008683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680803948675, max timestamp=1680803948686Num uniques 1068; 2023-04-06 17:59:08,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 17:59:08,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680803948675, max timestamp=1680803948686Num uniques 1068; 2023-04-06 17:59:08,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 17:59:08,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680803948675, max timestamp=1680803948686Num uniques 1068; 2023-04-06 17:59:08,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 17:59:08,710 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/ad9ddfe5e55f43e0a9b3ea3b6631d112 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ad9ddfe5e55f43e0a9b3ea3b6631d112 2023-04-06 17:59:08,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804008709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804008710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,714 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into ad9ddfe5e55f43e0a9b3ea3b6631d112(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:08,714 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:08,714 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=13, startTime=1680803947810; duration=0sec 2023-04-06 17:59:08,714 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:08,715 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/40a85d050da54bda8df1ec24bb71d4de as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/40a85d050da54bda8df1ec24bb71d4de 2023-04-06 17:59:08,718 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into 40a85d050da54bda8df1ec24bb71d4de(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:08,718 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:08,718 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=13, startTime=1680803947810; duration=0sec 2023-04-06 17:59:08,719 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:08,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804008787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804008787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804008788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804008813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:08,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804008813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,823 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/9963160703664fc09026205f0c25a3fd 2023-04-06 17:59:08,827 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/7568d31f320a4a84ba3410fc909a57a5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/7568d31f320a4a84ba3410fc909a57a5 2023-04-06 17:59:08,829 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/7568d31f320a4a84ba3410fc909a57a5, entries=150, sequenceid=176, filesize=25.3 K 2023-04-06 17:59:08,830 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/9ab7d7a6103c43bd85e36bcf7ca8823a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/9ab7d7a6103c43bd85e36bcf7ca8823a 2023-04-06 17:59:08,832 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/9ab7d7a6103c43bd85e36bcf7ca8823a, entries=150, sequenceid=176, filesize=11.8 K 2023-04-06 17:59:08,833 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/9963160703664fc09026205f0c25a3fd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9963160703664fc09026205f0c25a3fd 2023-04-06 17:59:08,836 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9963160703664fc09026205f0c25a3fd, entries=150, sequenceid=176, filesize=11.8 K 2023-04-06 17:59:08,836 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7893bef985668400d398f4c26aaccd85 in 890ms, sequenceid=176, compaction requested=false 2023-04-06 17:59:08,836 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:08,836 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:08,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:08,836 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:08,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:08,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:08,837 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,837 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:08,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,838 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b6cbf7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:08,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:08,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:08,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:08,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:08,840 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,840 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:08,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:08,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:08,840 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:08,842 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:08,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:08,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:08,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:08,842 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:08,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:08,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:08,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:08,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:08,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:08,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:08,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:08,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,846 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,846 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:08,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:08,846 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,946 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:08,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:08,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:08,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:08,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:08,949 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:08,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:08,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:08,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:08,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,949 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:08,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,950 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:08,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:08,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:08,950 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:08,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:08,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:08,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:08,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:08,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:08,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:08,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:08,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:08,956 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:08,956 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:08,956 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-06 17:59:08,956 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:08,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:08,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:08,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:08,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:08,957 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:08,978 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040659749c4261bd4fb5ac36218b2c60e347 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040659749c4261bd4fb5ac36218b2c60e347 2023-04-06 17:59:08,983 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=233, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/25006867e4514353add936c929053922 2023-04-06 17:59:08,991 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/af3159b984eb45eab4047ec383a80890 2023-04-06 17:59:08,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:08,993 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804008997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804008998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,003 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/33d6a5b1cb1943f4b28aac1296178cd0 2023-04-06 17:59:09,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804009001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,006 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/25006867e4514353add936c929053922 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/25006867e4514353add936c929053922 2023-04-06 17:59:09,009 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/25006867e4514353add936c929053922, entries=150, sequenceid=233, filesize=25.3 K 2023-04-06 17:59:09,009 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/af3159b984eb45eab4047ec383a80890 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/af3159b984eb45eab4047ec383a80890 2023-04-06 17:59:09,012 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/af3159b984eb45eab4047ec383a80890, entries=150, sequenceid=233, filesize=11.8 K 2023-04-06 17:59:09,012 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/33d6a5b1cb1943f4b28aac1296178cd0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/33d6a5b1cb1943f4b28aac1296178cd0 2023-04-06 17:59:09,016 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/33d6a5b1cb1943f4b28aac1296178cd0, entries=150, sequenceid=233, filesize=11.8 K 2023-04-06 17:59:09,017 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for 7893bef985668400d398f4c26aaccd85 in 61ms, sequenceid=233, compaction requested=true 2023-04-06 17:59:09,017 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:09,018 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,018 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,018 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,018 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,018 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,018 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:09,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:59:09,019 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,019 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:09,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:09,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:09,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:09,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:09,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,019 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,019 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9ffc2c[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,021 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,021 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,022 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,022 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,022 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,022 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,022 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,022 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,023 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,028 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,028 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,028 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,031 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,031 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,032 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,034 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,034 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,034 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803949041, max timestamp=1680803949054Num uniques 859; 2023-04-06 17:59:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:59:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680803949041, max timestamp=1680803949054Num uniques 859; 2023-04-06 17:59:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:59:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680803949041, max timestamp=1680803949054Num uniques 859; 2023-04-06 17:59:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 17:59:09,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804009089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804009089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804009102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804009102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804009105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,134 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,137 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,137 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,137 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,137 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,137 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,137 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,138 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,138 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,139 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,139 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,139 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,139 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,139 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,140 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,140 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,141 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,141 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d33d6c1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,142 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,142 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,143 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,143 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,143 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,143 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,144 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,147 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,147 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,147 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,147 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,147 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,148 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,148 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,148 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,152 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,152 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,152 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:09,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804009193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804009194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,252 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,255 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,255 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,256 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,256 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,256 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,256 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,257 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,257 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,257 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,257 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,258 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,258 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,258 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77fcac89[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,260 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,260 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,261 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,263 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,263 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,266 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,268 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,268 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,268 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,268 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804009304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804009305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804009309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,368 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,371 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,371 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,372 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,372 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,373 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,373 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,373 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,373 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,374 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,374 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@224038f2[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,375 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,376 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,376 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,377 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,379 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,379 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,379 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,380 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,380 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,381 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:09,383 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,382 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,383 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804009395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804009396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,432 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406d0492cdcf2a44f6f85103e3b8318a8ef to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d0492cdcf2a44f6f85103e3b8318a8ef 2023-04-06 17:59:09,436 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/ebb56e213c84462dbf9c23e814185dc4 2023-04-06 17:59:09,483 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,486 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,486 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,487 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,487 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,487 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,487 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,488 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,488 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,488 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,488 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,488 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,489 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,489 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33afde0a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,490 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,491 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,491 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,492 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,493 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,493 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,493 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,493 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,493 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,494 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,495 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,498 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:09,498 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,498 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,498 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,598 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,601 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,602 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,603 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,603 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,604 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,604 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,604 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,605 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@999360[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,606 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,607 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,607 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,607 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,608 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804009607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,609 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,609 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,610 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804009609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,611 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,611 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,611 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,612 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,613 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804009611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,614 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,614 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,615 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,615 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,615 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804009699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:09,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804009700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,715 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,717 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,717 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,717 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,717 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,718 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,718 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,719 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,719 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,719 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,720 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,720 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,730 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1293ef1e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,739 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,739 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,740 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,740 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,740 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,741 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,742 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,742 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,742 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,743 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,743 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,743 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,743 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,744 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,745 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,745 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,745 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,745 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,745 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,746 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,746 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,746 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,746 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,746 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,746 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,746 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,846 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/bb8f2bd4ead84f1dab6e506dc6949911 2023-04-06 17:59:09,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,848 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,851 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,851 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,851 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,852 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,852 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,852 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:09,852 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:09,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:09,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:09,852 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:09,853 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,853 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,853 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,853 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/5f26407ad620488f8d79320b37e6aca4 2023-04-06 17:59:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:09,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3137d216[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:09,855 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:09,855 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,855 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,855 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:09,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:09,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:09,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:09,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:09,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,856 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/ebb56e213c84462dbf9c23e814185dc4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ebb56e213c84462dbf9c23e814185dc4 2023-04-06 17:59:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:09,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:09,859 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:09,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:09,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:09,859 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:09,859 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:09,860 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,860 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,860 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,861 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:09,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ebb56e213c84462dbf9c23e814185dc4, entries=200, sequenceid=244, filesize=32.1 K 2023-04-06 17:59:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/bb8f2bd4ead84f1dab6e506dc6949911 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/bb8f2bd4ead84f1dab6e506dc6949911 2023-04-06 17:59:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,862 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,863 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,864 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:09,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:09,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,864 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:09,864 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:09,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:09,864 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:09,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/bb8f2bd4ead84f1dab6e506dc6949911, entries=150, sequenceid=244, filesize=11.8 K 2023-04-06 17:59:09,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/5f26407ad620488f8d79320b37e6aca4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f26407ad620488f8d79320b37e6aca4 2023-04-06 17:59:09,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f26407ad620488f8d79320b37e6aca4, entries=150, sequenceid=244, filesize=11.8 K 2023-04-06 17:59:09,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7893bef985668400d398f4c26aaccd85 in 851ms, sequenceid=244, compaction requested=true 2023-04-06 17:59:09,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:09,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:09,869 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:09,869 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:09,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:09,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:09,870 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110814 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:09,871 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:09,871 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,871 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/40a85d050da54bda8df1ec24bb71d4de, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/7568d31f320a4a84ba3410fc909a57a5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/25006867e4514353add936c929053922, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ebb56e213c84462dbf9c23e814185dc4] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=108.2 K 2023-04-06 17:59:09,871 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:09,871 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:09,871 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,871 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49909c88022b4591a3a836479a40c763, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/9ab7d7a6103c43bd85e36bcf7ca8823a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/af3159b984eb45eab4047ec383a80890, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/bb8f2bd4ead84f1dab6e506dc6949911] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=47.3 K 2023-04-06 17:59:09,872 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 49909c88022b4591a3a836479a40c763, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680803945417 2023-04-06 17:59:09,872 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 40a85d050da54bda8df1ec24bb71d4de, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680803945417 2023-04-06 17:59:09,872 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9ab7d7a6103c43bd85e36bcf7ca8823a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680803946535 2023-04-06 17:59:09,872 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 7568d31f320a4a84ba3410fc909a57a5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680803946535 2023-04-06 17:59:09,872 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 25006867e4514353add936c929053922, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803948702 2023-04-06 17:59:09,873 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting af3159b984eb45eab4047ec383a80890, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803948702 2023-04-06 17:59:09,873 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ebb56e213c84462dbf9c23e814185dc4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680803948993 2023-04-06 17:59:09,873 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting bb8f2bd4ead84f1dab6e506dc6949911, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680803949000 2023-04-06 17:59:09,878 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:09,884 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#529 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:09,893 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/370658efd6354a7e8c6e09c715988652 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/370658efd6354a7e8c6e09c715988652 2023-04-06 17:59:09,896 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into 370658efd6354a7e8c6e09c715988652(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:09,896 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:09,896 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=12, startTime=1680803949869; duration=0sec 2023-04-06 17:59:09,897 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:09,897 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:09,898 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:09,898 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:09,898 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,898 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ad9ddfe5e55f43e0a9b3ea3b6631d112, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9963160703664fc09026205f0c25a3fd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/33d6a5b1cb1943f4b28aac1296178cd0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f26407ad620488f8d79320b37e6aca4] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=47.3 K 2023-04-06 17:59:09,898 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting ad9ddfe5e55f43e0a9b3ea3b6631d112, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680803945417 2023-04-06 17:59:09,898 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9963160703664fc09026205f0c25a3fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680803946535 2023-04-06 17:59:09,899 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 33d6a5b1cb1943f4b28aac1296178cd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680803948702 2023-04-06 17:59:09,899 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5f26407ad620488f8d79320b37e6aca4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680803949000 2023-04-06 17:59:09,903 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:09,965 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:09,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:09,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:09,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:09,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:09,967 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:09,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:09,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:09,968 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,968 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,968 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:09,968 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:09,968 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:09,968 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:09,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:09,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:09,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:09,969 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:09,969 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:09,969 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:09,969 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:09,969 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:09,969 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:09,969 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:09,969 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-06 17:59:09,969 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:09,970 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:09,970 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:09,970 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:09,970 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:09,970 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:10,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:10,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:10,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804010120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804010123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804010124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804010203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804010208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57286 deadline: 1680804010224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804010226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804010226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,314 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/9c5fd69025df4ed781404ceb94a5643e as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9c5fd69025df4ed781404ceb94a5643e 2023-04-06 17:59:10,317 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into 9c5fd69025df4ed781404ceb94a5643e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:10,317 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:10,317 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=12, startTime=1680803949869; duration=0sec 2023-04-06 17:59:10,317 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:10,379 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406fd0716d6bc5540619eb27f3dcbd758dc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406fd0716d6bc5540619eb27f3dcbd758dc 2023-04-06 17:59:10,382 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/3e01afbdd7fa43dea6eeb4944dfefb94 2023-04-06 17:59:10,389 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/03e451ee704a4135a3e239f67347632d 2023-04-06 17:59:10,396 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/0d9cb715274349a4be9b2b680d7740d4 2023-04-06 17:59:10,398 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/3e01afbdd7fa43dea6eeb4944dfefb94 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3e01afbdd7fa43dea6eeb4944dfefb94 2023-04-06 17:59:10,400 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3e01afbdd7fa43dea6eeb4944dfefb94, entries=150, sequenceid=280, filesize=25.4 K 2023-04-06 17:59:10,401 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/03e451ee704a4135a3e239f67347632d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/03e451ee704a4135a3e239f67347632d 2023-04-06 17:59:10,403 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/03e451ee704a4135a3e239f67347632d, entries=150, sequenceid=280, filesize=11.9 K 2023-04-06 17:59:10,404 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/0d9cb715274349a4be9b2b680d7740d4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0d9cb715274349a4be9b2b680d7740d4 2023-04-06 17:59:10,406 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0d9cb715274349a4be9b2b680d7740d4, entries=150, sequenceid=280, filesize=11.9 K 2023-04-06 17:59:10,407 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 7893bef985668400d398f4c26aaccd85 in 438ms, sequenceid=280, compaction requested=false 2023-04-06 17:59:10,407 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:10,407 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,411 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:10,411 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:10,413 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,413 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,413 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,413 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:10,415 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:10,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca515b4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:10,415 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,415 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:10,415 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:10,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:10,416 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,416 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:10,416 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:10,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,416 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:10,418 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:10,418 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:10,418 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:10,418 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:10,418 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:10,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,422 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:10,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:10,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,424 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,425 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:10,425 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:10,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,425 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,425 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,424 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,425 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:10,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 17:59:10,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:10,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:10,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:10,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:10,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:10,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:10,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804010446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804010447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804010448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,451 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406466601c955024e14ba0e70f45b8c9d76 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406466601c955024e14ba0e70f45b8c9d76 2023-04-06 17:59:10,455 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=293, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/2b76cee71c9a473486cf73a466f17f7a 2023-04-06 17:59:10,525 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:10,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:10,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:10,527 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:10,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,527 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:10,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:10,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,528 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,528 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,529 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:10,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:10,530 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,530 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:10,530 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:10,530 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:10,530 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,530 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:10,530 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:10,530 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,530 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:10,530 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:10,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,531 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:10,531 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:10,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306b8bd3[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:10,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:10,532 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,533 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:10,533 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,533 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:10,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:10,533 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:10,534 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,534 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:10,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:10,535 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:10,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:10,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:10,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:10,535 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:10,536 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,537 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804010551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,554 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,554 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:10,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:10,555 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:10,555 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:10,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804010555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804010555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,559 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,559 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,655 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:10,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:10,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:10,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:10,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:10,658 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:10,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:10,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:10,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,659 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:10,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:10,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:10,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:10,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:10,660 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,660 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:10,661 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:10,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,661 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,662 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:10,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:10,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d8d69df[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:10,662 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,663 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:10,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:10,663 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:10,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,663 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:10,664 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,664 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,664 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,664 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:10,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:10,665 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:10,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:10,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:10,665 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:10,665 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:10,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,667 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,667 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,668 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:10,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:10,669 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,669 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:10,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,669 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:10,669 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,701 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/732db1ce2d1a4540981a8b332fa8a221 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/732db1ce2d1a4540981a8b332fa8a221 2023-04-06 17:59:10,704 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into 732db1ce2d1a4540981a8b332fa8a221(size=25.7 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:10,705 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:10,705 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=12, startTime=1680803949869; duration=0sec 2023-04-06 17:59:10,705 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:10,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804010754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804010758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:10,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804010758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,770 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:10,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:10,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:10,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:10,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:10,772 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:10,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:10,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:10,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,772 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:10,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:10,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:10,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:10,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:10,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:10,774 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:10,774 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:10,775 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,775 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,775 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,775 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:10,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3225bb1a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:10,776 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:10,776 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,778 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:10,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:10,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:10,778 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,778 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:10,778 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:10,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,778 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:10,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:10,780 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:10,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:10,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:10,780 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:10,780 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:10,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,781 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,781 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,783 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:10,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,784 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,784 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,784 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:10,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,784 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:10,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/e4c9f220aef949f39a2b64e2a49e9669 2023-04-06 17:59:10,885 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:10,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:10,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:10,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:10,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:10,887 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:10,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:10,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:10,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,889 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,889 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:10,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:10,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:10,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:10,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:10,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:10,889 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:10,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:10,890 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,890 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:10,890 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:10,890 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:10,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:10,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:10,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:10,890 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:10,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,891 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:10,892 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:10,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6679105f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:10,892 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,892 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:10,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:10,892 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:10,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,893 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:10,893 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:10,893 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,893 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,893 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:10,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:10,895 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:10,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:10,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:10,895 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:10,895 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:10,896 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,896 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:10,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,896 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,896 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:10,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,897 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:10,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:10,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:10,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:10,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:10,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:10,898 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:10,898 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:10,899 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:10,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:10,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:10,899 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,899 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:10,899 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:10,899 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:10,999 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:11,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:11,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:11,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:11,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:11,002 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:11,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:11,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:11,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:11,006 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:11,006 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,006 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:11,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:11,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:11,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:11,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:11,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:11,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:11,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:11,007 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,007 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:11,007 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:11,007 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:11,007 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:11,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:11,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:11,008 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,008 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,009 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:11,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa02a50[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:11,010 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:11,010 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,010 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:11,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:11,011 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:11,011 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,011 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:11,012 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:11,012 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,012 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:11,012 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:11,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:11,014 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:11,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:11,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:11,014 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:11,014 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:11,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,015 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,015 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,016 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,017 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,017 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,018 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:11,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:11,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:11,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:11,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:11,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:11,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:11,018 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:11,018 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,018 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:11,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:11,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804011058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804011059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804011062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,118 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:11,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:11,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:11,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:11,121 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:11,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:11,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:11,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,121 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:11,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:11,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:11,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:11,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:11,123 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,123 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:11,123 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:11,123 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:11,123 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:11,124 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,124 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,125 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:11,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb5794a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:11,126 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,126 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,127 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:11,127 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:11,127 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:11,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,127 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:11,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:11,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:11,128 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:11,128 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:11,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:11,130 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:11,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:11,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:11,130 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:11,130 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:11,132 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,132 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:11,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,132 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,132 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,134 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:11,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:11,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:11,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:11,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:11,135 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:11,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:11,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:11,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:11,135 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,136 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,136 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,136 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,136 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:11,136 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:11,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804011209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804011218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,236 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:11,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:11,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:11,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:11,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:11,239 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:11,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:11,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:11,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,239 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,240 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:11,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:11,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,240 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:11,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:11,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:11,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:11,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:11,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:11,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:11,242 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:11,243 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,243 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:11,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:11,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d7f4ad[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:11,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,245 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:11,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,245 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:11,245 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:11,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:11,246 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:11,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:11,246 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:11,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:11,248 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:11,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:11,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:11,248 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:11,248 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:11,248 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:11,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,249 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:11,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:11,249 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,250 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:11,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:11,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:11,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:11,251 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:11,251 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:11,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:11,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:11,251 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:11,251 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:11,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:11,252 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/4a663a327cea4481b39b65aa8fd4c293 2023-04-06 17:59:11,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/2b76cee71c9a473486cf73a466f17f7a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/2b76cee71c9a473486cf73a466f17f7a 2023-04-06 17:59:11,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/2b76cee71c9a473486cf73a466f17f7a, entries=150, sequenceid=293, filesize=25.4 K 2023-04-06 17:59:11,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/e4c9f220aef949f39a2b64e2a49e9669 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/e4c9f220aef949f39a2b64e2a49e9669 2023-04-06 17:59:11,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/e4c9f220aef949f39a2b64e2a49e9669, entries=150, sequenceid=293, filesize=11.9 K 2023-04-06 17:59:11,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/4a663a327cea4481b39b65aa8fd4c293 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4a663a327cea4481b39b65aa8fd4c293 2023-04-06 17:59:11,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4a663a327cea4481b39b65aa8fd4c293, entries=150, sequenceid=293, filesize=11.9 K 2023-04-06 17:59:11,281 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7893bef985668400d398f4c26aaccd85 in 852ms, sequenceid=293, compaction requested=true 2023-04-06 17:59:11,281 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:11,281 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:11,281 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:11,281 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:11,281 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:11,281 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:11,282 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78364 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:11,282 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:11,282 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:11,282 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:11,282 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,282 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,282 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/732db1ce2d1a4540981a8b332fa8a221, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3e01afbdd7fa43dea6eeb4944dfefb94, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/2b76cee71c9a473486cf73a466f17f7a] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=76.5 K 2023-04-06 17:59:11,282 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/370658efd6354a7e8c6e09c715988652, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/03e451ee704a4135a3e239f67347632d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/e4c9f220aef949f39a2b64e2a49e9669] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=35.9 K 2023-04-06 17:59:11,282 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 732db1ce2d1a4540981a8b332fa8a221, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680803949000 2023-04-06 17:59:11,282 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 370658efd6354a7e8c6e09c715988652, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680803949000 2023-04-06 17:59:11,283 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 3e01afbdd7fa43dea6eeb4944dfefb94, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680803949074 2023-04-06 17:59:11,283 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 03e451ee704a4135a3e239f67347632d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680803949074 2023-04-06 17:59:11,283 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 2b76cee71c9a473486cf73a466f17f7a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680803950122 2023-04-06 17:59:11,283 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting e4c9f220aef949f39a2b64e2a49e9669, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680803950122 2023-04-06 17:59:11,288 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#539 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:11,293 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#538 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:11,297 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/179f84c2425649d4b9577fb94ea7054f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/179f84c2425649d4b9577fb94ea7054f 2023-04-06 17:59:11,300 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into 179f84c2425649d4b9577fb94ea7054f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:11,300 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:11,300 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=13, startTime=1680803951281; duration=0sec 2023-04-06 17:59:11,300 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:11,300 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:11,300 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:11,301 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:11,301 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,301 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9c5fd69025df4ed781404ceb94a5643e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0d9cb715274349a4be9b2b680d7740d4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4a663a327cea4481b39b65aa8fd4c293] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=35.9 K 2023-04-06 17:59:11,301 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 9c5fd69025df4ed781404ceb94a5643e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680803949000 2023-04-06 17:59:11,301 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0d9cb715274349a4be9b2b680d7740d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680803949074 2023-04-06 17:59:11,301 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4a663a327cea4481b39b65aa8fd4c293, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680803950122 2023-04-06 17:59:11,307 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:11,351 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:11,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:11,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:11,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:11,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:11,354 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:11,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:11,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:11,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,354 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,355 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:11,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:11,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,355 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:11,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:11,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:11,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:11,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:11,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:11,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:11,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:11,356 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:11,356 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:11,356 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:11,356 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 17:59:11,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:11,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:11,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:11,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:11,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:11,357 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:11,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:11,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:11,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804011568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804011569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804011570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804011671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804011671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804011672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,714 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/a99bf21f129c405c95c79a2475077593 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a99bf21f129c405c95c79a2475077593 2023-04-06 17:59:11,717 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into a99bf21f129c405c95c79a2475077593(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:11,717 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:11,717 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=13, startTime=1680803951281; duration=0sec 2023-04-06 17:59:11,717 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:11,766 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304065889493263a84904ac96f4b5fd2acc03 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065889493263a84904ac96f4b5fd2acc03 2023-04-06 17:59:11,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804011873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804011875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:11,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:11,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804011875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,106 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/543554b5c77c44a6aec863cc7664252b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/543554b5c77c44a6aec863cc7664252b 2023-04-06 17:59:12,109 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into 543554b5c77c44a6aec863cc7664252b(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:12,109 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:12,109 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=13, startTime=1680803951281; duration=0sec 2023-04-06 17:59:12,109 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:12,170 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/e2b95bcfabf14e34ac2272bc156041b1 2023-04-06 17:59:12,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804012176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804012176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,177 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/c3235859f96c421385f50a6c9cbfb3dd 2023-04-06 17:59:12,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57346 deadline: 1680804012178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,584 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/331295e476ff45b9a90ba6596298a61c 2023-04-06 17:59:12,587 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/e2b95bcfabf14e34ac2272bc156041b1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e2b95bcfabf14e34ac2272bc156041b1 2023-04-06 17:59:12,589 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e2b95bcfabf14e34ac2272bc156041b1, entries=150, sequenceid=315, filesize=25.4 K 2023-04-06 17:59:12,590 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/c3235859f96c421385f50a6c9cbfb3dd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/c3235859f96c421385f50a6c9cbfb3dd 2023-04-06 17:59:12,592 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/c3235859f96c421385f50a6c9cbfb3dd, entries=150, sequenceid=315, filesize=11.9 K 2023-04-06 17:59:12,592 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/331295e476ff45b9a90ba6596298a61c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/331295e476ff45b9a90ba6596298a61c 2023-04-06 17:59:12,595 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/331295e476ff45b9a90ba6596298a61c, entries=150, sequenceid=315, filesize=11.9 K 2023-04-06 17:59:12,595 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7893bef985668400d398f4c26aaccd85 in 1239ms, sequenceid=315, compaction requested=false 2023-04-06 17:59:12,595 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:12,595 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:12,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:12,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:12,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:12,595 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:12,596 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,596 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:12,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:12,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d02ef60[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:12,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,598 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:12,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:12,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,598 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:12,598 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:12,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:12,599 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,599 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:12,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:12,601 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:12,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:12,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:12,601 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:12,601 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:12,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,602 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,603 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:12,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,605 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,605 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,605 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:12,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,605 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:12,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 17:59:12,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:12,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:12,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:12,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:12,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:12,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:12,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804012693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804012694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804012695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,705 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:12,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:12,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:12,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:12,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:12,708 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:12,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:12,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:12,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:12,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,708 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,709 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:12,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:12,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,709 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:12,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:12,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:12,713 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:12,713 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:12,713 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:12,713 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:12,713 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:12,723 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,723 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66fbba3a[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:12,724 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:12,725 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,729 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,729 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:12,729 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:12,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:12,729 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:12,730 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:12,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,730 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,730 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:12,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:12,732 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:12,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:12,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:12,732 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:12,732 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:12,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,732 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,733 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:12,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:12,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,735 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:12,735 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804012796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57334 deadline: 1680804012796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804012797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,835 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:12,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:12,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:12,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:12,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:12,838 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:12,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:12,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:12,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:12,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,838 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,839 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:12,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:12,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,839 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:12,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:12,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:12,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:12,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:12,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:12,840 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:12,841 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,841 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:12,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f3e086[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:12,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:12,842 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,843 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:12,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,843 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:12,844 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:12,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,844 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:12,846 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:12,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:12,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:12,846 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:12,846 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:12,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,847 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:12,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:12,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:12,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:12,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:12,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,851 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:12,851 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:12,951 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:12,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:12,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:12,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:12,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:12,953 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:12,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:12,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:12,953 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:12,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,954 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:12,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:12,954 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,955 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:12,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:12,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:12,956 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:12,956 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:12,956 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:12,956 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:12,958 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:12,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d7d0c89[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:12,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:12,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,960 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:12,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:12,961 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:12,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:12,963 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:12,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:12,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:12,963 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:12,963 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:12,964 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,964 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:12,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,964 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:12,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,966 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:12,967 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:12,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:12,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:12,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:12,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:12,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:12,968 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:12,968 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:13,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804013000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804013000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804013000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,068 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:13,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:13,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:13,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:13,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:13,071 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:13,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:13,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:13,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,071 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,072 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,072 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,072 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,072 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:13,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:13,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:13,073 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,073 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:13,073 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:13,073 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:13,074 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,074 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,075 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,075 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:13,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177e87a9[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:13,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:13,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:13,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:13,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:13,076 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:13,077 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:13,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:13,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:13,079 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:13,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:13,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:13,079 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:13,079 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:13,080 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,080 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,080 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,083 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:13,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,084 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,083 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,084 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:13,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:13,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,090 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068fa084c22bf948c0bb5cc10cebc2d199 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068fa084c22bf948c0bb5cc10cebc2d199 2023-04-06 17:59:13,095 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=330, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/129e912333c84f6088d15425d5f8a1af 2023-04-06 17:59:13,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/49860de4e8c248dc948dd45442023f7f 2023-04-06 17:59:13,184 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:13,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:13,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:13,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:13,187 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:13,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:13,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:13,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,187 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,188 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,188 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:13,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:13,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:13,189 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,189 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:13,189 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:13,189 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:13,189 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:13,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:13,190 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,190 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,191 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:13,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:13,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cefd516[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:13,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,193 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,193 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:13,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:13,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:13,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:13,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:13,194 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:13,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:13,196 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:13,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:13,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:13,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:13,196 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:13,197 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,198 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,199 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,199 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,200 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,200 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,200 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:13,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:13,201 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,200 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,201 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:13,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,201 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:13,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:13,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804013224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,226 DEBUG [Thread-3778] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:59:13,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804013236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,239 DEBUG [Thread-3780] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:59:13,301 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:13,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:13,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804013302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:13,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804013302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804013302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:13,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:13,304 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:13,304 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,304 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:13,304 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:13,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,305 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,305 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,306 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:13,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:13,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:13,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:13,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,306 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:13,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:13,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:13,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:13,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:13,307 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:13,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:13,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:13,307 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:13,308 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,308 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,308 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,308 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:13,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2296f7dd[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:13,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:13,309 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,310 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,310 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:13,310 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:13,310 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:13,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,310 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:13,311 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:13,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,311 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:13,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:13,313 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:13,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:13,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:13,313 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:13,313 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:13,313 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,313 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,314 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,314 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,315 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,316 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:13,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,316 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:13,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:13,316 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:13,316 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,317 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,417 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:13,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:13,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:13,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:13,419 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:13,419 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,419 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:13,419 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:13,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:13,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:13,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:13,422 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,422 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:13,422 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:13,422 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:13,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:13,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,423 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,423 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:13,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278abe42[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:13,424 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:13,424 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,425 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,425 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:13,425 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:13,425 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:13,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,425 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:13,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,426 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,426 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,426 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:13,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:13,427 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:13,427 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:13,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:13,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:13,428 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:13,428 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,428 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:13,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,428 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:13,429 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,430 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:13,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:13,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:13,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:13,432 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:13,432 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,432 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:13,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,432 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:13,432 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:13,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:13,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/a92787106d8b46ab900ec5e0ba265540 2023-04-06 17:59:13,516 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/129e912333c84f6088d15425d5f8a1af as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/129e912333c84f6088d15425d5f8a1af 2023-04-06 17:59:13,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/129e912333c84f6088d15425d5f8a1af, entries=150, sequenceid=330, filesize=25.4 K 2023-04-06 17:59:13,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/49860de4e8c248dc948dd45442023f7f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49860de4e8c248dc948dd45442023f7f 2023-04-06 17:59:13,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49860de4e8c248dc948dd45442023f7f, entries=150, sequenceid=330, filesize=11.9 K 2023-04-06 17:59:13,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/a92787106d8b46ab900ec5e0ba265540 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a92787106d8b46ab900ec5e0ba265540 2023-04-06 17:59:13,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a92787106d8b46ab900ec5e0ba265540, entries=150, sequenceid=330, filesize=11.9 K 2023-04-06 17:59:13,526 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7893bef985668400d398f4c26aaccd85 in 845ms, sequenceid=330, compaction requested=true 2023-04-06 17:59:13,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:13,526 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:13,526 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:13,526 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:13,526 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:13,526 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:13,526 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78616 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:13,527 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:13,527 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:13,527 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:13,527 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,527 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,527 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/543554b5c77c44a6aec863cc7664252b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e2b95bcfabf14e34ac2272bc156041b1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/129e912333c84f6088d15425d5f8a1af] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=76.8 K 2023-04-06 17:59:13,527 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/179f84c2425649d4b9577fb94ea7054f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/c3235859f96c421385f50a6c9cbfb3dd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49860de4e8c248dc948dd45442023f7f] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=36.2 K 2023-04-06 17:59:13,527 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 543554b5c77c44a6aec863cc7664252b, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680803950122 2023-04-06 17:59:13,527 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 179f84c2425649d4b9577fb94ea7054f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680803950122 2023-04-06 17:59:13,527 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting e2b95bcfabf14e34ac2272bc156041b1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680803950442 2023-04-06 17:59:13,528 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c3235859f96c421385f50a6c9cbfb3dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680803950442 2023-04-06 17:59:13,528 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 129e912333c84f6088d15425d5f8a1af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803951567 2023-04-06 17:59:13,528 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 49860de4e8c248dc948dd45442023f7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803951567 2023-04-06 17:59:13,532 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#548 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:13,532 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:13,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:13,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:13,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:13,538 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:13,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:13,538 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:13,539 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,539 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:13,540 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:13,540 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#547 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:13,541 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,541 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:13,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:13,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:13,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:13,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:13,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:13,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:13,546 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:13,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,546 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:13,547 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 17:59:13,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:13,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:13,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:13,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:13,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:13,547 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:13,549 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/76e6efcfd76c4099a65722524f3b0760 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/76e6efcfd76c4099a65722524f3b0760 2023-04-06 17:59:13,554 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into 76e6efcfd76c4099a65722524f3b0760(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:13,554 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:13,555 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=13, startTime=1680803953526; duration=0sec 2023-04-06 17:59:13,555 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:13,555 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:13,556 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:13,556 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:13,556 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:13,557 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a99bf21f129c405c95c79a2475077593, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/331295e476ff45b9a90ba6596298a61c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a92787106d8b46ab900ec5e0ba265540] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=36.2 K 2023-04-06 17:59:13,557 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a99bf21f129c405c95c79a2475077593, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680803950122 2023-04-06 17:59:13,557 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 331295e476ff45b9a90ba6596298a61c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680803950442 2023-04-06 17:59:13,558 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a92787106d8b46ab900ec5e0ba265540, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803951567 2023-04-06 17:59:13,575 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#550 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:13,577 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069720a554e5b34e69becc60504b457b28 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069720a554e5b34e69becc60504b457b28 2023-04-06 17:59:13,582 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/b6e532a22177428da449c0526401e06d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/b6e532a22177428da449c0526401e06d 2023-04-06 17:59:13,587 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into b6e532a22177428da449c0526401e06d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:13,587 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:13,587 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=13, startTime=1680803953526; duration=0sec 2023-04-06 17:59:13,587 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:13,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:13,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:13,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804013813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804013816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804013816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804013917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804013920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:13,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804013920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:13,971 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/90b2bfd1d4634a3088f4849dde1159a2 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/90b2bfd1d4634a3088f4849dde1159a2 2023-04-06 17:59:13,975 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into 90b2bfd1d4634a3088f4849dde1159a2(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:13,975 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:13,975 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=13, startTime=1680803953526; duration=0sec 2023-04-06 17:59:13,976 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:13,984 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=349, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/5f9eb440177a4fdab23eb5c80787a7ab 2023-04-06 17:59:14,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804014120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804014122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804014122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,392 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/121bae65260449518ea3e55522bfefa6 2023-04-06 17:59:14,402 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/f4b298867da242658bc53773a1adf41b 2023-04-06 17:59:14,406 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/5f9eb440177a4fdab23eb5c80787a7ab as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5f9eb440177a4fdab23eb5c80787a7ab 2023-04-06 17:59:14,408 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5f9eb440177a4fdab23eb5c80787a7ab, entries=150, sequenceid=349, filesize=25.4 K 2023-04-06 17:59:14,409 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/121bae65260449518ea3e55522bfefa6 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/121bae65260449518ea3e55522bfefa6 2023-04-06 17:59:14,413 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/121bae65260449518ea3e55522bfefa6, entries=150, sequenceid=349, filesize=11.9 K 2023-04-06 17:59:14,413 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/f4b298867da242658bc53773a1adf41b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/f4b298867da242658bc53773a1adf41b 2023-04-06 17:59:14,416 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/f4b298867da242658bc53773a1adf41b, entries=150, sequenceid=349, filesize=11.9 K 2023-04-06 17:59:14,417 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7893bef985668400d398f4c26aaccd85 in 870ms, sequenceid=349, compaction requested=false 2023-04-06 17:59:14,417 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:14,417 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:14,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:14,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:14,417 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:14,418 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,418 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,419 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:14,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44aa26d5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:14,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:14,420 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,420 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,421 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:14,421 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:14,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,421 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:14,421 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:14,421 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:14,421 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,421 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,422 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:14,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:14,424 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:14,424 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:14,424 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:14,424 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:14,424 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:14,426 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 17:59:14,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:14,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:14,426 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:14,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:14,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:14,427 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,428 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:14,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:14,431 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,431 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,431 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:14,431 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:14,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:14,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804014440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804014444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804014444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,449 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406907dbbda403242d8846c7c5ca7daf0b3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406907dbbda403242d8846c7c5ca7daf0b3 2023-04-06 17:59:14,532 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:14,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:14,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:14,534 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:14,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:14,534 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:14,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,535 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,535 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,536 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:14,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:14,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:14,537 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,537 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:14,537 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:14,537 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:14,537 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:14,538 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,538 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:14,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16bf1c33[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:14,539 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:14,539 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:14,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:14,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:14,540 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,540 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:14,540 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:14,540 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,541 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:14,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:14,542 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:14,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:14,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:14,542 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:14,542 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:14,543 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,543 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,543 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,543 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,545 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,546 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:14,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:14,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,546 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,547 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804014545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,546 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:14,547 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:14,547 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,547 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804014547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804014547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,647 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:14,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:14,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:14,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:14,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:14,649 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:14,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:14,649 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:14,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,650 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,650 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:14,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:14,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:14,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:14,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:14,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:14,651 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,651 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:14,651 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:14,652 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:14,652 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:14,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:14,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:14,652 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:14,652 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,653 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:14,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3823926[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:14,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:14,654 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,655 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:14,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,655 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:14,656 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,656 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:14,656 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,656 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:14,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:14,657 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:14,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:14,657 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:14,658 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:14,658 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:14,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,658 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,659 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,660 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,661 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,662 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:14,661 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,662 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:14,662 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804014749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804014750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804014750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,762 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:14,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:14,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:14,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:14,764 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:14,764 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,764 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:14,764 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:14,765 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,765 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,765 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,765 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,765 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,766 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:14,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:14,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:14,766 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:14,766 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,767 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:14,767 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:14,767 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:14,767 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:14,768 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,768 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,769 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:14,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7108ffba[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:14,769 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:14,769 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,770 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:14,770 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,770 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:14,771 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,771 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:14,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:14,773 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:14,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:14,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:14,773 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:14,773 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:14,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,773 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,774 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,775 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:14,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:14,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,776 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,776 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,776 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:14,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,776 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,856 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/6e24474adcf54a968eab7fd32b6df3e9 2023-04-06 17:59:14,876 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:14,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:14,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:14,879 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:14,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:14,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:14,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,879 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,880 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,880 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,880 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,880 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:14,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:14,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:14,881 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,881 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:14,881 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:14,881 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:14,881 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:14,882 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,882 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,883 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:14,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6401e3c0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:14,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:14,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,884 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,884 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:14,884 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,885 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:14,884 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:14,885 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:14,885 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:14,885 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,885 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,885 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:14,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:14,887 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:14,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:14,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:14,887 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:14,887 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:14,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,888 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,888 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,889 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,891 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:14,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:14,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:14,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,891 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,892 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,892 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:14,892 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:14,892 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:14,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:14,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:14,992 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:14,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:14,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:14,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:14,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:14,994 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:14,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:14,994 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:14,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,995 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:14,995 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:14,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,996 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:14,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:14,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:14,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:14,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:14,996 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:14,997 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,997 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:14,997 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:14,997 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:14,997 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:14,998 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:14,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:14,998 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:14,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:14,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a3125f7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:14,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:14,999 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,000 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:15,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,000 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:15,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:15,000 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:15,001 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,001 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:15,001 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,001 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:15,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:15,002 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:15,002 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:15,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:15,003 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:15,003 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:15,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,003 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,004 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,005 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:15,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:15,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,006 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:15,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:15,006 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,006 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:15,006 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,007 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804015053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:15,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804015054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:15,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804015055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,107 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:15,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:15,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:15,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:15,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:15,109 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:15,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:15,109 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:15,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,110 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,110 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:15,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:15,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:15,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:15,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:15,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:15,111 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,111 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:15,111 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:15,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:15,112 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:15,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:15,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:15,112 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:15,113 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,113 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:15,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e8769e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:15,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:15,114 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,114 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:15,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:15,115 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:15,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,115 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:15,116 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:15,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,116 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:15,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:15,118 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:15,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:15,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:15,118 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:15,118 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:15,119 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,119 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,119 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,119 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,120 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,122 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:15,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,122 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:15,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:15,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,122 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:15,122 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,223 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:15,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:15,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:15,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:15,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:15,225 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:15,225 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,225 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:15,225 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:15,226 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,226 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,226 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,226 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,227 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,227 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:15,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:15,228 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,228 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:15,228 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:15,228 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:15,228 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:15,229 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,229 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:15,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e2baf16[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:15,230 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:15,230 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,231 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,231 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:15,231 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:15,231 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:15,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,231 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:15,232 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:15,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,232 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:15,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:15,234 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:15,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:15,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:15,234 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:15,234 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:15,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,234 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,235 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,235 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,236 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,237 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:15,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,237 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:15,237 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:15,237 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/f63acf83a68f4cbe80cbc080a99c1bdd 2023-04-06 17:59:15,337 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:15,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:15,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:15,340 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:15,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:15,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:15,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,340 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,341 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,341 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:15,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:15,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:15,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:15,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:15,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:15,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:15,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:15,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:15,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:15,342 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:15,342 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:15,343 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,343 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:15,343 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:15,343 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,343 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,344 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:15,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45df56f0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:15,345 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:15,345 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,345 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:15,345 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,345 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:15,345 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:15,346 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:15,346 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,346 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:15,346 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,346 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:15,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:15,348 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:15,348 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:15,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:15,349 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:15,349 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:15,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,349 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,350 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,351 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,352 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:15,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:15,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,352 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,352 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,352 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,452 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:15,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:15,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:15,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:15,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:15,455 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:15,455 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,455 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:15,455 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:15,455 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,456 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,456 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:15,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:15,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:15,457 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:15,457 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,457 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:15,457 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:15,457 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:15,457 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,458 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:15,458 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:15,458 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:15,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:15,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:15,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:15,458 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,458 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,459 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,459 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:15,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c2cd21[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:15,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:15,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,460 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:15,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:15,460 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:15,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,460 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:15,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:15,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,461 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,461 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:15,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:15,463 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:15,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:15,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:15,463 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:15,463 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:15,464 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,464 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,464 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,465 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,466 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,467 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:15,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:15,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,467 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:15,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:15,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,467 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:15,467 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,468 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:15,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804015555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:15,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804015555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:15,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804015560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,568 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:15,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:15,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:15,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:15,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:15,570 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:15,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:15,570 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:15,571 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,571 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,571 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,571 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,572 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,572 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:15,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:15,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:15,572 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:15,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:15,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:15,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:15,573 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:15,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:15,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:15,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc50fe1[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:15,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:15,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,576 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:15,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:15,576 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,576 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:15,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:15,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:15,577 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:15,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:15,579 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:15,579 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:15,579 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:15,579 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:15,579 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:15,579 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,579 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:15,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,580 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:15,580 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,581 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:15,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:15,582 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:15,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:15,583 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,583 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:15,583 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:15,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:15,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,582 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:15,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:15,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/5f4260fc47da496393ed3900820460cc 2023-04-06 17:59:15,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/6e24474adcf54a968eab7fd32b6df3e9 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6e24474adcf54a968eab7fd32b6df3e9 2023-04-06 17:59:15,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6e24474adcf54a968eab7fd32b6df3e9, entries=200, sequenceid=367, filesize=32.3 K 2023-04-06 17:59:15,683 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:15,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:15,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/f63acf83a68f4cbe80cbc080a99c1bdd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f63acf83a68f4cbe80cbc080a99c1bdd 2023-04-06 17:59:15,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:15,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:15,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:15,685 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:15,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:15,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:15,685 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,686 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f63acf83a68f4cbe80cbc080a99c1bdd, entries=150, sequenceid=367, filesize=11.9 K 2023-04-06 17:59:15,686 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,686 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:15,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:15,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:15,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/5f4260fc47da496393ed3900820460cc as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f4260fc47da496393ed3900820460cc 2023-04-06 17:59:15,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:15,696 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:15,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:15,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:15,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:15,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:15,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:15,698 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:15,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f4260fc47da496393ed3900820460cc, entries=150, sequenceid=367, filesize=11.9 K 2023-04-06 17:59:15,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 7893bef985668400d398f4c26aaccd85 in 1273ms, sequenceid=367, compaction requested=true 2023-04-06 17:59:15,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:15,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:15,700 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:15,700 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:15,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:15,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:15,701 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,701 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,701 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/76e6efcfd76c4099a65722524f3b0760, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/121bae65260449518ea3e55522bfefa6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f63acf83a68f4cbe80cbc080a99c1bdd] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=36.3 K 2023-04-06 17:59:15,701 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/90b2bfd1d4634a3088f4849dde1159a2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5f9eb440177a4fdab23eb5c80787a7ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6e24474adcf54a968eab7fd32b6df3e9] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=83.7 K 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 76e6efcfd76c4099a65722524f3b0760, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803951567 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 90b2bfd1d4634a3088f4849dde1159a2, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803951567 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 121bae65260449518ea3e55522bfefa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680803952692 2023-04-06 17:59:15,701 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 5f9eb440177a4fdab23eb5c80787a7ab, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680803952692 2023-04-06 17:59:15,702 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f63acf83a68f4cbe80cbc080a99c1bdd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680803953809 2023-04-06 17:59:15,702 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 6e24474adcf54a968eab7fd32b6df3e9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680803953809 2023-04-06 17:59:15,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:15,702 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:15,702 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,702 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:15,702 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 17:59:15,702 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:15,702 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:15,703 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:15,703 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:15,703 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:15,703 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:15,712 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#557 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:15,724 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#556 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:15,731 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/6069100639124c12be56d9954bd8c2dd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6069100639124c12be56d9954bd8c2dd 2023-04-06 17:59:15,735 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into 6069100639124c12be56d9954bd8c2dd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:15,735 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:15,735 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=13, startTime=1680803955700; duration=0sec 2023-04-06 17:59:15,735 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:15,735 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:15,736 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:15,736 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:15,736 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:15,736 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/b6e532a22177428da449c0526401e06d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/f4b298867da242658bc53773a1adf41b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f4260fc47da496393ed3900820460cc] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=36.3 K 2023-04-06 17:59:15,737 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b6e532a22177428da449c0526401e06d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680803951567 2023-04-06 17:59:15,737 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting f4b298867da242658bc53773a1adf41b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680803952692 2023-04-06 17:59:15,737 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 5f4260fc47da496393ed3900820460cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680803953809 2023-04-06 17:59:15,742 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#559 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:16,139 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406e1c601190c1f45dab719c38d124b2e39 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e1c601190c1f45dab719c38d124b2e39 2023-04-06 17:59:16,143 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/d613746068a742b6b38ff46be22b1222 2023-04-06 17:59:16,150 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/7003c1a14fc74c62851cef04cdc7e524 2023-04-06 17:59:16,151 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/ba28c1f999f344b39495c297d717b5fd as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba28c1f999f344b39495c297d717b5fd 2023-04-06 17:59:16,154 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into ba28c1f999f344b39495c297d717b5fd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:16,154 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:16,154 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=13, startTime=1680803955700; duration=0sec 2023-04-06 17:59:16,154 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:16,557 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/a5a2b11b2d984609a8a983b34a41fb6d 2023-04-06 17:59:16,560 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/d613746068a742b6b38ff46be22b1222 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/d613746068a742b6b38ff46be22b1222 2023-04-06 17:59:16,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:16,563 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:16,563 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/d613746068a742b6b38ff46be22b1222, entries=150, sequenceid=385, filesize=25.4 K 2023-04-06 17:59:16,564 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/7003c1a14fc74c62851cef04cdc7e524 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7003c1a14fc74c62851cef04cdc7e524 2023-04-06 17:59:16,567 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7003c1a14fc74c62851cef04cdc7e524, entries=150, sequenceid=385, filesize=11.9 K 2023-04-06 17:59:16,567 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/a5a2b11b2d984609a8a983b34a41fb6d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a5a2b11b2d984609a8a983b34a41fb6d 2023-04-06 17:59:16,571 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a5a2b11b2d984609a8a983b34a41fb6d, entries=150, sequenceid=385, filesize=11.9 K 2023-04-06 17:59:16,572 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=26.84 KB/27480 for 7893bef985668400d398f4c26aaccd85 in 870ms, sequenceid=385, compaction requested=false 2023-04-06 17:59:16,572 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:16,572 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:16,572 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:16,572 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:16,572 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,572 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:16,572 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:16,573 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,573 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,574 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:16,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee55050[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:16,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:16,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,575 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,575 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:16,575 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:16,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:16,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:16,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 17:59:16,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:16,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:16,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:16,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:16,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:16,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:16,576 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:16,577 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:16,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,577 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:16,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:16,579 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:16,579 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:16,579 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:16,580 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:16,580 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:16,581 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,581 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,581 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,582 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,584 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:16,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:16,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:16,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:16,586 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:16,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,585 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,586 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:16,585 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,586 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,586 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,586 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680803956591, max timestamp=1680803956599Num uniques 859; 2023-04-06 17:59:16,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680803956591, max timestamp=1680803956599Num uniques 859; 2023-04-06 17:59:16,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680803956591, max timestamp=1680803956599Num uniques 859; 2023-04-06 17:59:16,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 17:59:16,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 17:59:16,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 17:59:16,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804016614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804016616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804016616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,686 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:16,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:16,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:16,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:16,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:16,688 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:16,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:16,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:16,688 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:16,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,689 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:16,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:16,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,689 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:16,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:16,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:16,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:16,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:16,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:16,690 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:16,691 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:16,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:16,691 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:16,691 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:16,691 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:16,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:16,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:16,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:16,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:16,691 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,691 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,692 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,692 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:16,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b2f3aa[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:16,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:16,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,694 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:16,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:16,694 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:16,694 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:16,695 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,695 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:16,695 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,695 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:16,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:16,697 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:16,697 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:16,697 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:16,697 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:16,697 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:16,698 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,698 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,698 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,698 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,699 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:16,701 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:16,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:16,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:16,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:16,701 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:16,702 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,702 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:16,702 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,702 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:16,702 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,702 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804016717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804016719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804016719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,802 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:16,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:16,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:16,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:16,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:16,804 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:16,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:16,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:16,804 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:16,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,805 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:16,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:16,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,805 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:16,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:16,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:16,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:16,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:16,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:16,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:16,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:16,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:16,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:16,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:16,806 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:16,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:16,806 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:16,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:16,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:16,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,807 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,807 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:16,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c98778[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:16,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:16,808 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,809 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:16,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,809 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:16,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:16,809 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:16,810 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:16,810 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,810 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,810 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:16,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:16,811 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:16,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:16,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:16,811 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:16,811 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:16,812 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,812 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,812 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,812 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,813 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:16,814 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:16,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:16,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:16,814 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:16,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,815 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,815 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:16,815 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,815 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:16,815 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,815 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,915 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:16,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:16,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:16,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:16,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:16,918 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:16,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:16,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:16,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:16,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:16,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:16,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:16,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:16,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:16,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:16,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:16,920 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:16,921 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,921 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,921 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,921 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:16,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c63aa7[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:16,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:16,922 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804016921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804016921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,923 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,923 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:16,923 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:16,923 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,923 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:16,923 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:16,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:16,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804016922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,924 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,924 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:16,924 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,924 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:16,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:16,926 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:16,926 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:16,926 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:16,926 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:16,926 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:16,927 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:16,927 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,927 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:16,927 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,928 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:16,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:16,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,929 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:16,930 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:16,930 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:16,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:16,930 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:16,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:16,930 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:16,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:16,930 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:16,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:16,930 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:16,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:16,942 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/c3ac1a97524b45ff88a291b494cda0a4 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/c3ac1a97524b45ff88a291b494cda0a4 2023-04-06 17:59:16,946 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into c3ac1a97524b45ff88a291b494cda0a4(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 17:59:16,946 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:16,946 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=13, startTime=1680803955700; duration=1sec 2023-04-06 17:59:16,946 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:16,987 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b6beef3fa69b4b1e95204651d7f3d080 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b6beef3fa69b4b1e95204651d7f3d080 2023-04-06 17:59:17,030 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,033 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,033 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,034 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,034 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,034 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,034 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,035 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,035 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,035 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,035 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,036 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,036 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,037 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,037 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70659bbf[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,038 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,038 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,039 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,039 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,039 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,039 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,039 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,039 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,040 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,040 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,040 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,040 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,042 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,042 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,042 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,042 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,042 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,043 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,043 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,043 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,043 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,044 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,046 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,046 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,046 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,046 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,146 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,149 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,149 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,150 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,150 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,151 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,151 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,151 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,151 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,151 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,151 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,151 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,151 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,151 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,152 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,153 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,153 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360c19ba[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,155 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,157 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,157 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,158 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,159 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,160 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,161 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,161 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804017225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804017225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804017225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57330 deadline: 1680804017256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,258 DEBUG [Thread-3778] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:59:17,261 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,263 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,267 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,268 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,268 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,268 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,268 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,268 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,269 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,269 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bb3aa44[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,271 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,271 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,271 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,271 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,271 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,272 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57310 deadline: 1680804017271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,272 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,272 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,273 DEBUG [Thread-3780] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., hostname=jenkins-hbase20.apache.org,38773,1680803776439, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 17:59:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,275 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,275 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,275 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,276 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,276 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,277 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,279 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,279 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,279 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,279 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,379 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,382 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,382 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,383 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,383 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,384 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,384 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,385 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,385 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,386 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e25b497[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,387 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,387 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,387 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,387 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,388 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,388 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,388 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,389 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,391 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,391 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,391 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,391 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=397, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/1758803dc18c4cb884b697e82447dc26 2023-04-06 17:59:17,392 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,392 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,392 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,393 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,395 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,396 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,397 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,397 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,397 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,397 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,497 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,500 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,500 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,500 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,501 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,501 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,502 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,502 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,503 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,503 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,504 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,504 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22c1708f[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,505 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,505 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,505 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,505 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,505 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,506 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,506 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,508 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,508 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,508 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,509 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,509 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,511 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,512 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,512 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,512 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,512 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,513 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,612 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,616 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,616 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,617 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,617 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,618 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,618 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,618 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,618 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,618 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,619 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,619 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74566ca4[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,620 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,620 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,620 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,621 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,621 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,621 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,621 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,622 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,623 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,623 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,623 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,624 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,624 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,624 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,624 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,625 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,626 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,626 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,626 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,626 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,727 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804017727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804017728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,729 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,729 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,730 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,730 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,731 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:17,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804017729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,731 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,732 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,732 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,732 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,732 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,733 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,733 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,733 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,733 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125119e0[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,734 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,734 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,734 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,734 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,735 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,735 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,737 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,737 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,737 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,737 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,737 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,738 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,738 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,739 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,741 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,741 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,741 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,741 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/6404df08040d4cae812e1b0969df27aa 2023-04-06 17:59:17,841 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,844 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,844 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,845 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,845 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,846 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,847 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,847 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,848 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41039631[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,849 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,850 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,850 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,850 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,851 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,851 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,851 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,851 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,853 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,853 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,853 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,853 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,853 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,854 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,854 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,854 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,854 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,855 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,856 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,857 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,857 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,856 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,857 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,957 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:17,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:17,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:17,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:17,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:17,959 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:17,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:17,959 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:17,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,960 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,960 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,961 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:17,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:17,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:17,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:17,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:17,961 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:17,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:17,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:17,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:17,962 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:17,963 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,963 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,963 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,963 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:17,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68efc396[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:17,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:17,964 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,964 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,965 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,965 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:17,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:17,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:17,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:17,965 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,965 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,966 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:17,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:17,967 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:17,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:17,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:17,967 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:17,967 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:17,968 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,968 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,968 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,969 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:17,970 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:17,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:17,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:17,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:17,972 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:17,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:17,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:17,971 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,972 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:17,972 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:17,972 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:17,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:17,972 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:17,972 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,972 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:17,972 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:17,972 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,072 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:18,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:18,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:18,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:18,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:18,075 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:18,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:18,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:18,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:18,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,075 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,076 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:18,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:18,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,076 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:18,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:18,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:18,077 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,077 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:18,077 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:18,077 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:18,077 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,078 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:18,078 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:18,078 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,078 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:18,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:18,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f9bb163[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:18,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:18,080 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,080 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,080 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:18,081 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:18,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,081 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:18,081 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:18,081 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:18,081 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:18,081 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:18,081 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:18,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:18,083 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:18,084 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:18,084 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:18,084 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:18,084 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:18,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:18,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,084 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:18,085 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:18,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,086 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:18,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:18,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:18,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:18,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,087 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,088 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,087 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:18,088 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:18,088 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:18,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:18,188 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:18,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:18,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:18,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:18,190 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:18,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:18,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:18,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:18,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,191 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,191 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:18,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:18,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,192 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:18,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:18,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:18,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:18,193 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:18,193 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:18,193 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:18,193 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:18,193 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-06 17:59:18,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,194 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:18,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,194 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:18,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:18,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@359f3369[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:18,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:18,196 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,197 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:18,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:18,197 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:18,197 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,197 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:18,198 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:18,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:18,198 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:18,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:18,199 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:18,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:18,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:18,199 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:18,199 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:18,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:18,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,201 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:18,201 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,202 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:18,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:18,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:18,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,203 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:18,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:18,203 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:18,204 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:18,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:18,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:18,204 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,204 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:18,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/4f25da7a791848a2888091511a367c59 2023-04-06 17:59:18,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/1758803dc18c4cb884b697e82447dc26 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/1758803dc18c4cb884b697e82447dc26 2023-04-06 17:59:18,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/1758803dc18c4cb884b697e82447dc26, entries=350, sequenceid=397, filesize=52.9 K 2023-04-06 17:59:18,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/6404df08040d4cae812e1b0969df27aa as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6404df08040d4cae812e1b0969df27aa 2023-04-06 17:59:18,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6404df08040d4cae812e1b0969df27aa, entries=150, sequenceid=397, filesize=11.9 K 2023-04-06 17:59:18,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/4f25da7a791848a2888091511a367c59 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4f25da7a791848a2888091511a367c59 2023-04-06 17:59:18,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4f25da7a791848a2888091511a367c59, entries=150, sequenceid=397, filesize=11.9 K 2023-04-06 17:59:18,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 7893bef985668400d398f4c26aaccd85 in 1641ms, sequenceid=397, compaction requested=true 2023-04-06 17:59:18,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:18,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:18,217 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:18,217 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:18,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:18,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:18,218 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:18,218 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:18,218 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:18,218 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:18,218 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,218 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,218 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/c3ac1a97524b45ff88a291b494cda0a4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/d613746068a742b6b38ff46be22b1222, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/1758803dc18c4cb884b697e82447dc26] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=104.5 K 2023-04-06 17:59:18,218 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6069100639124c12be56d9954bd8c2dd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7003c1a14fc74c62851cef04cdc7e524, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6404df08040d4cae812e1b0969df27aa] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=36.4 K 2023-04-06 17:59:18,219 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting c3ac1a97524b45ff88a291b494cda0a4, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680803953809 2023-04-06 17:59:18,219 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6069100639124c12be56d9954bd8c2dd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680803953809 2023-04-06 17:59:18,219 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting d613746068a742b6b38ff46be22b1222, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680803954437 2023-04-06 17:59:18,219 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7003c1a14fc74c62851cef04cdc7e524, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680803954437 2023-04-06 17:59:18,219 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 1758803dc18c4cb884b697e82447dc26, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680803956562 2023-04-06 17:59:18,219 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 6404df08040d4cae812e1b0969df27aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680803956563 2023-04-06 17:59:18,223 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#566 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:18,227 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#565 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:18,304 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:18,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:18,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:18,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:18,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:18,307 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:18,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:18,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:18,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:18,307 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,308 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:18,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:18,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,308 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:18,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:18,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:18,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:18,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:18,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:18,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:18,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:18,309 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:18,310 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:18,310 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 17:59:18,319 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040653c02d0437da44599c1117ff101c6a6b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040653c02d0437da44599c1117ff101c6a6b 2023-04-06 17:59:18,629 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/0191c1a31dae46088faf48d2c779c661 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0191c1a31dae46088faf48d2c779c661 2023-04-06 17:59:18,642 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into 0191c1a31dae46088faf48d2c779c661(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:18,642 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:18,642 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=13, startTime=1680803958217; duration=0sec 2023-04-06 17:59:18,642 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:18,642 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 17:59:18,643 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 17:59:18,643 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:18,643 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:18,643 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba28c1f999f344b39495c297d717b5fd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a5a2b11b2d984609a8a983b34a41fb6d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4f25da7a791848a2888091511a367c59] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=36.4 K 2023-04-06 17:59:18,643 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting ba28c1f999f344b39495c297d717b5fd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680803953809 2023-04-06 17:59:18,644 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting a5a2b11b2d984609a8a983b34a41fb6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680803954437 2023-04-06 17:59:18,644 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 4f25da7a791848a2888091511a367c59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680803956563 2023-04-06 17:59:18,645 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/5b77f934debb44aabfba78b34c39024d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5b77f934debb44aabfba78b34c39024d 2023-04-06 17:59:18,648 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#568 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:18,648 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into 5b77f934debb44aabfba78b34c39024d(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:18,648 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:18,648 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=13, startTime=1680803958217; duration=0sec 2023-04-06 17:59:18,648 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:18,653 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/bef331f9281d486d9e68981095a2ab86 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/bef331f9281d486d9e68981095a2ab86 2023-04-06 17:59:18,656 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into bef331f9281d486d9e68981095a2ab86(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:18,656 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:18,656 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=13, startTime=1680803958218; duration=0sec 2023-04-06 17:59:18,656 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:18,723 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=434, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/b3d3a9d29b634bf3aea7bb9d439cb7ff 2023-04-06 17:59:18,729 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/7456331296aa4c6b8cda9e5a6c140c7a 2023-04-06 17:59:18,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:18,732 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:18,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:18,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804018736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:18,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:18,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804018738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804018738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:18,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804018838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:18,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804018842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:18,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:18,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804018842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804019042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804019043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804019044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,137 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/c1caad50fb3b43d2b2717c5d291234e0 2023-04-06 17:59:19,140 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/b3d3a9d29b634bf3aea7bb9d439cb7ff as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/b3d3a9d29b634bf3aea7bb9d439cb7ff 2023-04-06 17:59:19,143 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/b3d3a9d29b634bf3aea7bb9d439cb7ff, entries=150, sequenceid=434, filesize=25.4 K 2023-04-06 17:59:19,144 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/7456331296aa4c6b8cda9e5a6c140c7a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7456331296aa4c6b8cda9e5a6c140c7a 2023-04-06 17:59:19,146 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7456331296aa4c6b8cda9e5a6c140c7a, entries=150, sequenceid=434, filesize=11.9 K 2023-04-06 17:59:19,147 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/c1caad50fb3b43d2b2717c5d291234e0 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c1caad50fb3b43d2b2717c5d291234e0 2023-04-06 17:59:19,149 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c1caad50fb3b43d2b2717c5d291234e0, entries=150, sequenceid=434, filesize=11.9 K 2023-04-06 17:59:19,150 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 7893bef985668400d398f4c26aaccd85 in 840ms, sequenceid=434, compaction requested=false 2023-04-06 17:59:19,150 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:19,150 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:19,151 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:19,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,152 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:19,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:19,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f089f5[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:19,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:19,153 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:19,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:19,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:19,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,154 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:19,154 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:19,154 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:19,155 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:19,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:19,156 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:19,156 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:19,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:19,157 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:19,157 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:19,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:19,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,157 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:19,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:19,158 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:19,159 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:19,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:19,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:19,160 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:19,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:19,160 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:19,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:19,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:19,161 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:19,161 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:19,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:19,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:19,161 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,261 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:19,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:19,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 17:59:19,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 17:59:19,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 17:59:19,263 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 17:59:19,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:19,263 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 17:59:19,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 17:59:19,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:19,264 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:19,264 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:19,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,265 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 17:59:19,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 17:59:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:19,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 17:59:19,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 17:59:19,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 17:59:19,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 17:59:19,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 17:59:19,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 17:59:19,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:19,266 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. started... 2023-04-06 17:59:19,266 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:19,266 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:19,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:19,348 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. as already flushing 2023-04-06 17:59:19,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680803959356, max timestamp=1680803959361Num uniques 859; 2023-04-06 17:59:19,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680803959356, max timestamp=1680803959361Num uniques 859; 2023-04-06 17:59:19,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 17:59:19,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680803959356, max timestamp=1680803959361Num uniques 859; 2023-04-06 17:59:19,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 17:59:19,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 17:59:19,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804019380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804019380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804019381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804019484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804019485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804019485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,676 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406460bb1f68c194b6d9d85075451d656d6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406460bb1f68c194b6d9d85075451d656d6 2023-04-06 17:59:19,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804019687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804019687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804019688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804019989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804019990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:19,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:19,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804019990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,080 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/ddd549f6838d4be2985dacd0084b964c 2023-04-06 17:59:20,488 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/2405baf670de43d480d54c52168e9f6f 2023-04-06 17:59:20,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:20,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57346 deadline: 1680804020494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:20,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57286 deadline: 1680804020494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 17:59:20,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38773] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57334 deadline: 1680804020496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,896 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/240514e7082f4e4f812c589487a884c5 2023-04-06 17:59:20,898 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/ddd549f6838d4be2985dacd0084b964c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ddd549f6838d4be2985dacd0084b964c 2023-04-06 17:59:20,901 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ddd549f6838d4be2985dacd0084b964c, entries=150, sequenceid=446, filesize=25.4 K 2023-04-06 17:59:20,902 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/2405baf670de43d480d54c52168e9f6f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/2405baf670de43d480d54c52168e9f6f 2023-04-06 17:59:20,905 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/2405baf670de43d480d54c52168e9f6f, entries=150, sequenceid=446, filesize=11.9 K 2023-04-06 17:59:20,905 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/240514e7082f4e4f812c589487a884c5 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/240514e7082f4e4f812c589487a884c5 2023-04-06 17:59:20,908 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/240514e7082f4e4f812c589487a884c5, entries=150, sequenceid=446, filesize=11.9 K 2023-04-06 17:59:20,909 INFO [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7893bef985668400d398f4c26aaccd85 in 1643ms, sequenceid=446, compaction requested=true 2023-04-06 17:59:20,909 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:20,909 DEBUG [rs(jenkins-hbase20.apache.org,38773,1680803776439)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 17:59:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 17:59:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 17:59:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 17:59:20,910 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 17:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38773,1680803776439' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 17:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e15040e[Count = 0] remaining members to acquire global barrier 2023-04-06 17:59:20,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 17:59:20,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,912 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 17:59:20,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 17:59:20,912 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38773,1680803776439' in zk 2023-04-06 17:59:20,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,912 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 17:59:20,913 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 17:59:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38773,1680803776439' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 17:59:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38773,1680803776439': 2023-04-06 17:59:20,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38773,1680803776439' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 17:59:20,915 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 17:59:20,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 17:59:20,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 17:59:20,916 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 17:59:20,916 INFO [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 17:59:20,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:20,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:20,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:20,916 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 17:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 17:59:20,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 17:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:20,917 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 17:59:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:20,918 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 17:59:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 17:59:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:20,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 17:59:20,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 17:59:20,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 17:59:20,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 17:59:20,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 17:59:20,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 17:59:20,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 17:59:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 17:59:20,919 DEBUG [(jenkins-hbase20.apache.org,40911,1680803775368)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 17:59:20,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:20,919 DEBUG [Thread-3782] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 17:59:20,919 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,920 DEBUG [Thread-3782] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 17:59:20,920 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 17:59:20,920 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 17:59:20,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 17:59:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 17:59:20,954 DEBUG [Thread-3787] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00fcbd84 to 127.0.0.1:49384 2023-04-06 17:59:20,954 DEBUG [Thread-3787] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:20,955 DEBUG [Thread-3785] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27acfde4 to 127.0.0.1:49384 2023-04-06 17:59:20,955 DEBUG [Thread-3785] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:20,957 DEBUG [Thread-3791] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ef34de3 to 127.0.0.1:49384 2023-04-06 17:59:20,957 DEBUG [Thread-3791] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:20,957 DEBUG [Thread-3783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x768816e9 to 127.0.0.1:49384 2023-04-06 17:59:20,957 DEBUG [Thread-3783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:20,958 DEBUG [Thread-3789] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18884189 to 127.0.0.1:49384 2023-04-06 17:59:20,958 DEBUG [Thread-3789] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:21,020 DEBUG [Thread-3782] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 17:59:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 17:59:21,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38773] regionserver.HRegion(9158): Flush requested on 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:21,501 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-06 17:59:21,502 DEBUG [Thread-3774] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x082ccebe to 127.0.0.1:49384 2023-04-06 17:59:21,502 DEBUG [Thread-3774] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:21,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:21,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:21,504 DEBUG [Thread-3776] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5768f049 to 127.0.0.1:49384 2023-04-06 17:59:21,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:21,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:21,504 DEBUG [Thread-3776] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:21,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:21,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 17:59:21,507 DEBUG [Thread-3772] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c2ce5cd to 127.0.0.1:49384 2023-04-06 17:59:21,507 DEBUG [Thread-3772] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:21,922 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304063911f4bc1cfc4deba9efafd5d89740f8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063911f4bc1cfc4deba9efafd5d89740f8 2023-04-06 17:59:22,330 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=484, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/210f5241228647b692b409fe612feb4f 2023-04-06 17:59:22,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/b8304dde3e42459886dc91eaca44f2fb 2023-04-06 17:59:23,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/ba74d7b6db7c4752b998a364ec618a01 2023-04-06 17:59:23,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/210f5241228647b692b409fe612feb4f as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/210f5241228647b692b409fe612feb4f 2023-04-06 17:59:23,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/210f5241228647b692b409fe612feb4f, entries=150, sequenceid=484, filesize=25.4 K 2023-04-06 17:59:23,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/b8304dde3e42459886dc91eaca44f2fb as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/b8304dde3e42459886dc91eaca44f2fb 2023-04-06 17:59:23,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/b8304dde3e42459886dc91eaca44f2fb, entries=150, sequenceid=484, filesize=11.9 K 2023-04-06 17:59:23,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/ba74d7b6db7c4752b998a364ec618a01 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba74d7b6db7c4752b998a364ec618a01 2023-04-06 17:59:23,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba74d7b6db7c4752b998a364ec618a01, entries=150, sequenceid=484, filesize=11.9 K 2023-04-06 17:59:23,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=6.71 KB/6870 for 7893bef985668400d398f4c26aaccd85 in 1668ms, sequenceid=484, compaction requested=true 2023-04-06 17:59:23,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:23,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:23,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:23,169 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:23,169 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:23,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:23,171 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:23,171 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/A is initiating minor compaction (all files) 2023-04-06 17:59:23,171 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:23,171 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/A in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:23,171 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/B is initiating minor compaction (all files) 2023-04-06 17:59:23,171 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5b77f934debb44aabfba78b34c39024d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/b3d3a9d29b634bf3aea7bb9d439cb7ff, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ddd549f6838d4be2985dacd0084b964c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/210f5241228647b692b409fe612feb4f] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=102.5 K 2023-04-06 17:59:23,171 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/B in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:23,171 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0191c1a31dae46088faf48d2c779c661, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7456331296aa4c6b8cda9e5a6c140c7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/2405baf670de43d480d54c52168e9f6f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/b8304dde3e42459886dc91eaca44f2fb] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=48.4 K 2023-04-06 17:59:23,171 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 5b77f934debb44aabfba78b34c39024d, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680803956563 2023-04-06 17:59:23,171 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 0191c1a31dae46088faf48d2c779c661, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680803956563 2023-04-06 17:59:23,172 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting b3d3a9d29b634bf3aea7bb9d439cb7ff, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680803956613 2023-04-06 17:59:23,172 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 7456331296aa4c6b8cda9e5a6c140c7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680803956613 2023-04-06 17:59:23,172 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting ddd549f6838d4be2985dacd0084b964c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680803958735 2023-04-06 17:59:23,172 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 2405baf670de43d480d54c52168e9f6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680803958735 2023-04-06 17:59:23,172 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] compactions.Compactor(207): Compacting 210f5241228647b692b409fe612feb4f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680803959376 2023-04-06 17:59:23,172 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting b8304dde3e42459886dc91eaca44f2fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680803959376 2023-04-06 17:59:23,181 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#B#compaction#578 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:23,183 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#A#compaction#577 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:23,650 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/40fd26a2fa704a2ab7dc8de25118cb2c as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/40fd26a2fa704a2ab7dc8de25118cb2c 2023-04-06 17:59:23,656 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/B of 7893bef985668400d398f4c26aaccd85 into 40fd26a2fa704a2ab7dc8de25118cb2c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:23,656 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:23,656 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/B, priority=12, startTime=1680803963169; duration=0sec 2023-04-06 17:59:23,656 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 17:59:23,656 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 17:59:23,657 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/e4fe1417874c4e20926ac6f64142ea8b as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e4fe1417874c4e20926ac6f64142ea8b 2023-04-06 17:59:23,658 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 17:59:23,658 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1912): 7893bef985668400d398f4c26aaccd85/C is initiating minor compaction (all files) 2023-04-06 17:59:23,658 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7893bef985668400d398f4c26aaccd85/C in TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:23,658 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/bef331f9281d486d9e68981095a2ab86, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c1caad50fb3b43d2b2717c5d291234e0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/240514e7082f4e4f812c589487a884c5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba74d7b6db7c4752b998a364ec618a01] into tmpdir=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp, totalSize=48.4 K 2023-04-06 17:59:23,658 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting bef331f9281d486d9e68981095a2ab86, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680803956563 2023-04-06 17:59:23,659 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting c1caad50fb3b43d2b2717c5d291234e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680803956613 2023-04-06 17:59:23,659 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting 240514e7082f4e4f812c589487a884c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680803958735 2023-04-06 17:59:23,659 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] compactions.Compactor(207): Compacting ba74d7b6db7c4752b998a364ec618a01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680803959376 2023-04-06 17:59:23,661 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/A of 7893bef985668400d398f4c26aaccd85 into e4fe1417874c4e20926ac6f64142ea8b(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:23,661 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:23,661 INFO [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/A, priority=12, startTime=1680803963169; duration=0sec 2023-04-06 17:59:23,661 DEBUG [RS:0;jenkins-hbase20:38773-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:23,666 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] throttle.PressureAwareThroughputController(145): 7893bef985668400d398f4c26aaccd85#C#compaction#579 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 17:59:24,081 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/2df8880802a24946822bf59e0fe72c65 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2df8880802a24946822bf59e0fe72c65 2023-04-06 17:59:24,086 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7893bef985668400d398f4c26aaccd85/C of 7893bef985668400d398f4c26aaccd85 into 2df8880802a24946822bf59e0fe72c65(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 17:59:24,086 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:24,087 INFO [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85., storeName=7893bef985668400d398f4c26aaccd85/C, priority=12, startTime=1680803963169; duration=0sec 2023-04-06 17:59:24,087 DEBUG [RS:0;jenkins-hbase20:38773-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 17:59:27,314 DEBUG [Thread-3780] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x451cdd43 to 127.0.0.1:49384 2023-04-06 17:59:27,314 DEBUG [Thread-3780] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:27,332 DEBUG [Thread-3778] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67b5579f to 127.0.0.1:49384 2023-04-06 17:59:27,332 DEBUG [Thread-3778] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 84 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 5982 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 6019 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 6141 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 5985 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(398): read 6077 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 17:59:27,332 INFO [Listener at localhost.localdomain/40169] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 17:59:27,333 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25d2db45 to 127.0.0.1:49384 2023-04-06 17:59:27,333 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:27,333 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 17:59:27,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 17:59:27,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:27,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 17:59:27,338 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803967337"}]},"ts":"1680803967337"} 2023-04-06 17:59:27,339 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 17:59:27,364 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 17:59:27,365 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, UNASSIGN}] 2023-04-06 17:59:27,366 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, UNASSIGN 2023-04-06 17:59:27,367 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:27,367 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803967367"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680803967367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680803967367"}]},"ts":"1680803967367"} 2023-04-06 17:59:27,369 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439}] 2023-04-06 17:59:27,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 17:59:27,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:27,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 7893bef985668400d398f4c26aaccd85, disabling compactions & flushes 2023-04-06 17:59:27,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:27,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:27,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. after waiting 0 ms 2023-04-06 17:59:27,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:27,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 7893bef985668400d398f4c26aaccd85 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 17:59:27,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=A 2023-04-06 17:59:27,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:27,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=B 2023-04-06 17:59:27,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:27,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7893bef985668400d398f4c26aaccd85, store=C 2023-04-06 17:59:27,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 17:59:27,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ad09135ae9164b90a37dd85048164717 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ad09135ae9164b90a37dd85048164717 2023-04-06 17:59:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 17:59:27,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 17:59:27,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=493, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/6025ee9d110e49c6a62a0467c9c1353a 2023-04-06 17:59:28,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/6615aa36870d4e4b8acdd7d422545d2a 2023-04-06 17:59:28,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/c5d4bbad1370415b9bd39d930b93396d 2023-04-06 17:59:28,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/A/6025ee9d110e49c6a62a0467c9c1353a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6025ee9d110e49c6a62a0467c9c1353a 2023-04-06 17:59:28,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6025ee9d110e49c6a62a0467c9c1353a, entries=150, sequenceid=493, filesize=25.4 K 2023-04-06 17:59:28,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/B/6615aa36870d4e4b8acdd7d422545d2a as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6615aa36870d4e4b8acdd7d422545d2a 2023-04-06 17:59:28,397 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6615aa36870d4e4b8acdd7d422545d2a, entries=150, sequenceid=493, filesize=11.9 K 2023-04-06 17:59:28,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/.tmp/C/c5d4bbad1370415b9bd39d930b93396d as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c5d4bbad1370415b9bd39d930b93396d 2023-04-06 17:59:28,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c5d4bbad1370415b9bd39d930b93396d, entries=150, sequenceid=493, filesize=11.9 K 2023-04-06 17:59:28,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 7893bef985668400d398f4c26aaccd85 in 881ms, sequenceid=493, compaction requested=false 2023-04-06 17:59:28,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ea90f483d33f4ffd9315c92a4e062174, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3da143f0255f48d3a2c369d2cfaaaea2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/43c884a30b594249ad433f806121f8d6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5ef77f34540b45aeb6d939939bec97bd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/cb045c6e4174404a99a15141fb3272f2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/940e9aefd79a4bb5adeb2cba506c29d2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/40a85d050da54bda8df1ec24bb71d4de, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/35dce7d7f5e04f45a4dd46262947847c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/7568d31f320a4a84ba3410fc909a57a5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/25006867e4514353add936c929053922, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ebb56e213c84462dbf9c23e814185dc4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/732db1ce2d1a4540981a8b332fa8a221, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3e01afbdd7fa43dea6eeb4944dfefb94, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/543554b5c77c44a6aec863cc7664252b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/2b76cee71c9a473486cf73a466f17f7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e2b95bcfabf14e34ac2272bc156041b1, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/90b2bfd1d4634a3088f4849dde1159a2, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/129e912333c84f6088d15425d5f8a1af, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5f9eb440177a4fdab23eb5c80787a7ab, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6e24474adcf54a968eab7fd32b6df3e9, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/c3ac1a97524b45ff88a291b494cda0a4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/d613746068a742b6b38ff46be22b1222, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/1758803dc18c4cb884b697e82447dc26, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5b77f934debb44aabfba78b34c39024d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/b3d3a9d29b634bf3aea7bb9d439cb7ff, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ddd549f6838d4be2985dacd0084b964c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/210f5241228647b692b409fe612feb4f] to archive 2023-04-06 17:59:28,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:59:28,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ea90f483d33f4ffd9315c92a4e062174 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ea90f483d33f4ffd9315c92a4e062174 2023-04-06 17:59:28,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3da143f0255f48d3a2c369d2cfaaaea2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3da143f0255f48d3a2c369d2cfaaaea2 2023-04-06 17:59:28,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/43c884a30b594249ad433f806121f8d6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/43c884a30b594249ad433f806121f8d6 2023-04-06 17:59:28,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5ef77f34540b45aeb6d939939bec97bd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5ef77f34540b45aeb6d939939bec97bd 2023-04-06 17:59:28,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/cb045c6e4174404a99a15141fb3272f2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/cb045c6e4174404a99a15141fb3272f2 2023-04-06 17:59:28,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/940e9aefd79a4bb5adeb2cba506c29d2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/940e9aefd79a4bb5adeb2cba506c29d2 2023-04-06 17:59:28,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/40a85d050da54bda8df1ec24bb71d4de to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/40a85d050da54bda8df1ec24bb71d4de 2023-04-06 17:59:28,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/35dce7d7f5e04f45a4dd46262947847c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/35dce7d7f5e04f45a4dd46262947847c 2023-04-06 17:59:28,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/7568d31f320a4a84ba3410fc909a57a5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/7568d31f320a4a84ba3410fc909a57a5 2023-04-06 17:59:28,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/25006867e4514353add936c929053922 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/25006867e4514353add936c929053922 2023-04-06 17:59:28,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ebb56e213c84462dbf9c23e814185dc4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ebb56e213c84462dbf9c23e814185dc4 2023-04-06 17:59:28,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/732db1ce2d1a4540981a8b332fa8a221 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/732db1ce2d1a4540981a8b332fa8a221 2023-04-06 17:59:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 17:59:28,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3e01afbdd7fa43dea6eeb4944dfefb94 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/3e01afbdd7fa43dea6eeb4944dfefb94 2023-04-06 17:59:28,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/543554b5c77c44a6aec863cc7664252b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/543554b5c77c44a6aec863cc7664252b 2023-04-06 17:59:28,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/2b76cee71c9a473486cf73a466f17f7a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/2b76cee71c9a473486cf73a466f17f7a 2023-04-06 17:59:28,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e2b95bcfabf14e34ac2272bc156041b1 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e2b95bcfabf14e34ac2272bc156041b1 2023-04-06 17:59:28,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/90b2bfd1d4634a3088f4849dde1159a2 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/90b2bfd1d4634a3088f4849dde1159a2 2023-04-06 17:59:28,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/129e912333c84f6088d15425d5f8a1af to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/129e912333c84f6088d15425d5f8a1af 2023-04-06 17:59:28,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5f9eb440177a4fdab23eb5c80787a7ab to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5f9eb440177a4fdab23eb5c80787a7ab 2023-04-06 17:59:28,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6e24474adcf54a968eab7fd32b6df3e9 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6e24474adcf54a968eab7fd32b6df3e9 2023-04-06 17:59:28,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/c3ac1a97524b45ff88a291b494cda0a4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/c3ac1a97524b45ff88a291b494cda0a4 2023-04-06 17:59:28,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/d613746068a742b6b38ff46be22b1222 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/d613746068a742b6b38ff46be22b1222 2023-04-06 17:59:28,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/1758803dc18c4cb884b697e82447dc26 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/1758803dc18c4cb884b697e82447dc26 2023-04-06 17:59:28,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5b77f934debb44aabfba78b34c39024d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/5b77f934debb44aabfba78b34c39024d 2023-04-06 17:59:28,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/b3d3a9d29b634bf3aea7bb9d439cb7ff to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/b3d3a9d29b634bf3aea7bb9d439cb7ff 2023-04-06 17:59:28,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ddd549f6838d4be2985dacd0084b964c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/ddd549f6838d4be2985dacd0084b964c 2023-04-06 17:59:28,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/210f5241228647b692b409fe612feb4f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/210f5241228647b692b409fe612feb4f 2023-04-06 17:59:28,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/816db5813a8f4dfaa4c7699a5ad70b95, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f692f75e9b6d48adaf4f75d7c95ab714, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/28c35d570da14a39ae607b6ed409a809, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/a0e9fda13e6e411d8a3e32706dd901ba, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/5e4d3e564dc449ed949d14f2c882eb59, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/4433f00aa02e4d2ab94fadc97bcd2089, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49909c88022b4591a3a836479a40c763, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0a85122ed58c489299e186067fa69d02, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/9ab7d7a6103c43bd85e36bcf7ca8823a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/af3159b984eb45eab4047ec383a80890, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/370658efd6354a7e8c6e09c715988652, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/bb8f2bd4ead84f1dab6e506dc6949911, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/03e451ee704a4135a3e239f67347632d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/179f84c2425649d4b9577fb94ea7054f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/e4c9f220aef949f39a2b64e2a49e9669, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/c3235859f96c421385f50a6c9cbfb3dd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/76e6efcfd76c4099a65722524f3b0760, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49860de4e8c248dc948dd45442023f7f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/121bae65260449518ea3e55522bfefa6, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6069100639124c12be56d9954bd8c2dd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f63acf83a68f4cbe80cbc080a99c1bdd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7003c1a14fc74c62851cef04cdc7e524, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0191c1a31dae46088faf48d2c779c661, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6404df08040d4cae812e1b0969df27aa, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7456331296aa4c6b8cda9e5a6c140c7a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/2405baf670de43d480d54c52168e9f6f, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/b8304dde3e42459886dc91eaca44f2fb] to archive 2023-04-06 17:59:28,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:59:28,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/816db5813a8f4dfaa4c7699a5ad70b95 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/816db5813a8f4dfaa4c7699a5ad70b95 2023-04-06 17:59:28,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f692f75e9b6d48adaf4f75d7c95ab714 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f692f75e9b6d48adaf4f75d7c95ab714 2023-04-06 17:59:28,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/28c35d570da14a39ae607b6ed409a809 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/28c35d570da14a39ae607b6ed409a809 2023-04-06 17:59:28,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/a0e9fda13e6e411d8a3e32706dd901ba to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/a0e9fda13e6e411d8a3e32706dd901ba 2023-04-06 17:59:28,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/5e4d3e564dc449ed949d14f2c882eb59 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/5e4d3e564dc449ed949d14f2c882eb59 2023-04-06 17:59:28,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/4433f00aa02e4d2ab94fadc97bcd2089 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/4433f00aa02e4d2ab94fadc97bcd2089 2023-04-06 17:59:28,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49909c88022b4591a3a836479a40c763 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49909c88022b4591a3a836479a40c763 2023-04-06 17:59:28,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0a85122ed58c489299e186067fa69d02 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0a85122ed58c489299e186067fa69d02 2023-04-06 17:59:28,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/9ab7d7a6103c43bd85e36bcf7ca8823a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/9ab7d7a6103c43bd85e36bcf7ca8823a 2023-04-06 17:59:28,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/af3159b984eb45eab4047ec383a80890 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/af3159b984eb45eab4047ec383a80890 2023-04-06 17:59:28,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/370658efd6354a7e8c6e09c715988652 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/370658efd6354a7e8c6e09c715988652 2023-04-06 17:59:28,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/bb8f2bd4ead84f1dab6e506dc6949911 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/bb8f2bd4ead84f1dab6e506dc6949911 2023-04-06 17:59:28,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/03e451ee704a4135a3e239f67347632d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/03e451ee704a4135a3e239f67347632d 2023-04-06 17:59:28,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/179f84c2425649d4b9577fb94ea7054f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/179f84c2425649d4b9577fb94ea7054f 2023-04-06 17:59:28,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/e4c9f220aef949f39a2b64e2a49e9669 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/e4c9f220aef949f39a2b64e2a49e9669 2023-04-06 17:59:28,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/c3235859f96c421385f50a6c9cbfb3dd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/c3235859f96c421385f50a6c9cbfb3dd 2023-04-06 17:59:28,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/76e6efcfd76c4099a65722524f3b0760 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/76e6efcfd76c4099a65722524f3b0760 2023-04-06 17:59:28,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49860de4e8c248dc948dd45442023f7f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/49860de4e8c248dc948dd45442023f7f 2023-04-06 17:59:28,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/121bae65260449518ea3e55522bfefa6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/121bae65260449518ea3e55522bfefa6 2023-04-06 17:59:28,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6069100639124c12be56d9954bd8c2dd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6069100639124c12be56d9954bd8c2dd 2023-04-06 17:59:28,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f63acf83a68f4cbe80cbc080a99c1bdd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/f63acf83a68f4cbe80cbc080a99c1bdd 2023-04-06 17:59:28,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7003c1a14fc74c62851cef04cdc7e524 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7003c1a14fc74c62851cef04cdc7e524 2023-04-06 17:59:28,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0191c1a31dae46088faf48d2c779c661 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/0191c1a31dae46088faf48d2c779c661 2023-04-06 17:59:28,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6404df08040d4cae812e1b0969df27aa to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6404df08040d4cae812e1b0969df27aa 2023-04-06 17:59:28,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7456331296aa4c6b8cda9e5a6c140c7a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/7456331296aa4c6b8cda9e5a6c140c7a 2023-04-06 17:59:28,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/2405baf670de43d480d54c52168e9f6f to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/2405baf670de43d480d54c52168e9f6f 2023-04-06 17:59:28,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/b8304dde3e42459886dc91eaca44f2fb to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/b8304dde3e42459886dc91eaca44f2fb 2023-04-06 17:59:28,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2a9c3806ba5545bb93b94c1de595c8f8, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/97c8b30e8be844d1bdd99ffd3e89ec88, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/810e3c80baf24e67af99b67d95a82cb3, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/6e8cf84dbc324b6e9ee0a72c7c5d0dce, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/85fc2597550a4f5880e951e8c6d0410b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0c73689c3f544ea69db477751d51827a, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ad9ddfe5e55f43e0a9b3ea3b6631d112, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/fc71e4e449be4e928f99345689f94844, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9963160703664fc09026205f0c25a3fd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/33d6a5b1cb1943f4b28aac1296178cd0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9c5fd69025df4ed781404ceb94a5643e, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f26407ad620488f8d79320b37e6aca4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0d9cb715274349a4be9b2b680d7740d4, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a99bf21f129c405c95c79a2475077593, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4a663a327cea4481b39b65aa8fd4c293, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/331295e476ff45b9a90ba6596298a61c, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/b6e532a22177428da449c0526401e06d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a92787106d8b46ab900ec5e0ba265540, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/f4b298867da242658bc53773a1adf41b, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba28c1f999f344b39495c297d717b5fd, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f4260fc47da496393ed3900820460cc, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a5a2b11b2d984609a8a983b34a41fb6d, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/bef331f9281d486d9e68981095a2ab86, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4f25da7a791848a2888091511a367c59, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c1caad50fb3b43d2b2717c5d291234e0, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/240514e7082f4e4f812c589487a884c5, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba74d7b6db7c4752b998a364ec618a01] to archive 2023-04-06 17:59:28,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 17:59:28,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2a9c3806ba5545bb93b94c1de595c8f8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2a9c3806ba5545bb93b94c1de595c8f8 2023-04-06 17:59:28,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/97c8b30e8be844d1bdd99ffd3e89ec88 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/97c8b30e8be844d1bdd99ffd3e89ec88 2023-04-06 17:59:28,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/810e3c80baf24e67af99b67d95a82cb3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/810e3c80baf24e67af99b67d95a82cb3 2023-04-06 17:59:28,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/6e8cf84dbc324b6e9ee0a72c7c5d0dce to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/6e8cf84dbc324b6e9ee0a72c7c5d0dce 2023-04-06 17:59:28,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/85fc2597550a4f5880e951e8c6d0410b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/85fc2597550a4f5880e951e8c6d0410b 2023-04-06 17:59:28,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0c73689c3f544ea69db477751d51827a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0c73689c3f544ea69db477751d51827a 2023-04-06 17:59:28,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ad9ddfe5e55f43e0a9b3ea3b6631d112 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ad9ddfe5e55f43e0a9b3ea3b6631d112 2023-04-06 17:59:28,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/fc71e4e449be4e928f99345689f94844 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/fc71e4e449be4e928f99345689f94844 2023-04-06 17:59:28,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9963160703664fc09026205f0c25a3fd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9963160703664fc09026205f0c25a3fd 2023-04-06 17:59:28,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/33d6a5b1cb1943f4b28aac1296178cd0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/33d6a5b1cb1943f4b28aac1296178cd0 2023-04-06 17:59:28,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9c5fd69025df4ed781404ceb94a5643e to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/9c5fd69025df4ed781404ceb94a5643e 2023-04-06 17:59:28,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f26407ad620488f8d79320b37e6aca4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f26407ad620488f8d79320b37e6aca4 2023-04-06 17:59:28,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0d9cb715274349a4be9b2b680d7740d4 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/0d9cb715274349a4be9b2b680d7740d4 2023-04-06 17:59:28,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a99bf21f129c405c95c79a2475077593 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a99bf21f129c405c95c79a2475077593 2023-04-06 17:59:28,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4a663a327cea4481b39b65aa8fd4c293 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4a663a327cea4481b39b65aa8fd4c293 2023-04-06 17:59:28,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/331295e476ff45b9a90ba6596298a61c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/331295e476ff45b9a90ba6596298a61c 2023-04-06 17:59:28,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/b6e532a22177428da449c0526401e06d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/b6e532a22177428da449c0526401e06d 2023-04-06 17:59:28,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a92787106d8b46ab900ec5e0ba265540 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a92787106d8b46ab900ec5e0ba265540 2023-04-06 17:59:28,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/f4b298867da242658bc53773a1adf41b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/f4b298867da242658bc53773a1adf41b 2023-04-06 17:59:28,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba28c1f999f344b39495c297d717b5fd to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba28c1f999f344b39495c297d717b5fd 2023-04-06 17:59:28,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f4260fc47da496393ed3900820460cc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/5f4260fc47da496393ed3900820460cc 2023-04-06 17:59:28,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a5a2b11b2d984609a8a983b34a41fb6d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/a5a2b11b2d984609a8a983b34a41fb6d 2023-04-06 17:59:28,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/bef331f9281d486d9e68981095a2ab86 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/bef331f9281d486d9e68981095a2ab86 2023-04-06 17:59:28,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4f25da7a791848a2888091511a367c59 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/4f25da7a791848a2888091511a367c59 2023-04-06 17:59:28,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c1caad50fb3b43d2b2717c5d291234e0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c1caad50fb3b43d2b2717c5d291234e0 2023-04-06 17:59:28,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/240514e7082f4e4f812c589487a884c5 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/240514e7082f4e4f812c589487a884c5 2023-04-06 17:59:28,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba74d7b6db7c4752b998a364ec618a01 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/ba74d7b6db7c4752b998a364ec618a01 2023-04-06 17:59:28,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/recovered.edits/496.seqid, newMaxSeqId=496, maxSeqId=4 2023-04-06 17:59:28,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85. 2023-04-06 17:59:28,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 7893bef985668400d398f4c26aaccd85: 2023-04-06 17:59:28,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:28,533 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=7893bef985668400d398f4c26aaccd85, regionState=CLOSED 2023-04-06 17:59:28,533 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680803968533"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680803968533"}]},"ts":"1680803968533"} 2023-04-06 17:59:28,535 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-06 17:59:28,536 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 7893bef985668400d398f4c26aaccd85, server=jenkins-hbase20.apache.org,38773,1680803776439 in 1.1650 sec 2023-04-06 17:59:28,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-06 17:59:28,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7893bef985668400d398f4c26aaccd85, UNASSIGN in 1.1710 sec 2023-04-06 17:59:28,537 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680803968537"}]},"ts":"1680803968537"} 2023-04-06 17:59:28,538 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 17:59:28,540 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 17:59:28,541 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2060 sec 2023-04-06 17:59:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 17:59:29,444 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-06 17:59:29,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 17:59:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:29,449 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:29,450 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:29,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-06 17:59:29,459 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:29,463 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C, FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/recovered.edits] 2023-04-06 17:59:29,469 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6025ee9d110e49c6a62a0467c9c1353a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/6025ee9d110e49c6a62a0467c9c1353a 2023-04-06 17:59:29,472 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e4fe1417874c4e20926ac6f64142ea8b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/A/e4fe1417874c4e20926ac6f64142ea8b 2023-04-06 17:59:29,477 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/40fd26a2fa704a2ab7dc8de25118cb2c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/40fd26a2fa704a2ab7dc8de25118cb2c 2023-04-06 17:59:29,479 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6615aa36870d4e4b8acdd7d422545d2a to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/B/6615aa36870d4e4b8acdd7d422545d2a 2023-04-06 17:59:29,482 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2df8880802a24946822bf59e0fe72c65 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/2df8880802a24946822bf59e0fe72c65 2023-04-06 17:59:29,483 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c5d4bbad1370415b9bd39d930b93396d to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/C/c5d4bbad1370415b9bd39d930b93396d 2023-04-06 17:59:29,486 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/recovered.edits/496.seqid to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85/recovered.edits/496.seqid 2023-04-06 17:59:29,487 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/.tmp/data/default/TestAcidGuarantees/7893bef985668400d398f4c26aaccd85 2023-04-06 17:59:29,487 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 17:59:29,487 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 17:59:29,488 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 17:59:29,492 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406297249748d3f4fd597294ddf6a6d2124 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406297249748d3f4fd597294ddf6a6d2124 2023-04-06 17:59:29,494 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063911f4bc1cfc4deba9efafd5d89740f8 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063911f4bc1cfc4deba9efafd5d89740f8 2023-04-06 17:59:29,495 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406460bb1f68c194b6d9d85075451d656d6 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406460bb1f68c194b6d9d85075451d656d6 2023-04-06 17:59:29,497 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406466601c955024e14ba0e70f45b8c9d76 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406466601c955024e14ba0e70f45b8c9d76 2023-04-06 17:59:29,499 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040653c02d0437da44599c1117ff101c6a6b to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040653c02d0437da44599c1117ff101c6a6b 2023-04-06 17:59:29,501 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065889493263a84904ac96f4b5fd2acc03 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065889493263a84904ac96f4b5fd2acc03 2023-04-06 17:59:29,503 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040659749c4261bd4fb5ac36218b2c60e347 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040659749c4261bd4fb5ac36218b2c60e347 2023-04-06 17:59:29,504 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068fa084c22bf948c0bb5cc10cebc2d199 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068fa084c22bf948c0bb5cc10cebc2d199 2023-04-06 17:59:29,506 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406907dbbda403242d8846c7c5ca7daf0b3 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406907dbbda403242d8846c7c5ca7daf0b3 2023-04-06 17:59:29,507 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040691c507f058e845da8c08ba6cb29c108c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040691c507f058e845da8c08ba6cb29c108c 2023-04-06 17:59:29,508 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069720a554e5b34e69becc60504b457b28 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069720a554e5b34e69becc60504b457b28 2023-04-06 17:59:29,509 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ad09135ae9164b90a37dd85048164717 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ad09135ae9164b90a37dd85048164717 2023-04-06 17:59:29,509 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b5b2fb4a835a4aeb8b8ed59623a49c45 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b5b2fb4a835a4aeb8b8ed59623a49c45 2023-04-06 17:59:29,510 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b6beef3fa69b4b1e95204651d7f3d080 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b6beef3fa69b4b1e95204651d7f3d080 2023-04-06 17:59:29,511 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c2ee244e28324aae929b9de563089014 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c2ee244e28324aae929b9de563089014 2023-04-06 17:59:29,512 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c9d9a2bc2ea346dd80173c8cbab687d0 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c9d9a2bc2ea346dd80173c8cbab687d0 2023-04-06 17:59:29,512 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d0492cdcf2a44f6f85103e3b8318a8ef to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d0492cdcf2a44f6f85103e3b8318a8ef 2023-04-06 17:59:29,513 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ddf0dc700cbf46a98c8c6731bb005015 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ddf0dc700cbf46a98c8c6731bb005015 2023-04-06 17:59:29,514 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e1c601190c1f45dab719c38d124b2e39 to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e1c601190c1f45dab719c38d124b2e39 2023-04-06 17:59:29,515 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e295c7d6eab242e4b6e6a6e6a58fcf6c to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e295c7d6eab242e4b6e6a6e6a58fcf6c 2023-04-06 17:59:29,515 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406fd0716d6bc5540619eb27f3dcbd758dc to hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406fd0716d6bc5540619eb27f3dcbd758dc 2023-04-06 17:59:29,516 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 17:59:29,517 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:29,519 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 17:59:29,520 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 17:59:29,521 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:29,521 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 17:59:29,521 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680803969521"}]},"ts":"9223372036854775807"} 2023-04-06 17:59:29,523 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 17:59:29,523 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7893bef985668400d398f4c26aaccd85, NAME => 'TestAcidGuarantees,,1680803937795.7893bef985668400d398f4c26aaccd85.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 17:59:29,523 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 17:59:29,523 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680803969523"}]},"ts":"9223372036854775807"} 2023-04-06 17:59:29,525 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 17:59:29,526 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 17:59:29,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 81 msec 2023-04-06 17:59:29,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-06 17:59:29,552 INFO [Listener at localhost.localdomain/40169] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-06 17:59:29,571 INFO [Listener at localhost.localdomain/40169] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=532 (was 543), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=313 (was 343), ProcessCount=167 (was 167), AvailableMemoryMB=3938 (was 3816) - AvailableMemoryMB LEAK? - 2023-04-06 17:59:29,572 INFO [Listener at localhost.localdomain/40169] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-06 17:59:29,572 INFO [Listener at localhost.localdomain/40169] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 17:59:29,573 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3eb02786 to 127.0.0.1:49384 2023-04-06 17:59:29,573 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:29,573 DEBUG [Listener at localhost.localdomain/40169] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-06 17:59:29,573 DEBUG [Listener at localhost.localdomain/40169] util.JVMClusterUtil(257): Found active master hash=194421377, stopped=false 2023-04-06 17:59:29,573 INFO [Listener at localhost.localdomain/40169] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:59:29,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-06 17:59:29,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-06 17:59:29,574 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:59:29,574 INFO [Listener at localhost.localdomain/40169] procedure2.ProcedureExecutor(629): Stopping 2023-04-06 17:59:29,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 17:59:29,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 17:59:29,574 DEBUG [Listener at localhost.localdomain/40169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a2c2b34 to 127.0.0.1:49384 2023-04-06 17:59:29,574 DEBUG [Listener at localhost.localdomain/40169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:29,574 INFO [Listener at localhost.localdomain/40169] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,38773,1680803776439' ***** 2023-04-06 17:59:29,574 INFO [Listener at localhost.localdomain/40169] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-06 17:59:29,575 INFO [RS:0;jenkins-hbase20:38773] regionserver.HeapMemoryManager(220): Stopping 2023-04-06 17:59:29,575 INFO [RS:0;jenkins-hbase20:38773] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-06 17:59:29,575 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-06 17:59:29,575 INFO [RS:0;jenkins-hbase20:38773] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-06 17:59:29,575 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(3304): Received CLOSE for 0d49654bd69ee321ab98d34d9a593b8e 2023-04-06 17:59:29,575 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:29,575 DEBUG [RS:0;jenkins-hbase20:38773] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3b6661 to 127.0.0.1:49384 2023-04-06 17:59:29,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 0d49654bd69ee321ab98d34d9a593b8e, disabling compactions & flushes 2023-04-06 17:59:29,576 DEBUG [RS:0;jenkins-hbase20:38773] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:29,576 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:59:29,576 INFO [RS:0;jenkins-hbase20:38773] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-06 17:59:29,576 INFO [RS:0;jenkins-hbase20:38773] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-06 17:59:29,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:59:29,576 INFO [RS:0;jenkins-hbase20:38773] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-06 17:59:29,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. after waiting 0 ms 2023-04-06 17:59:29,576 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-06 17:59:29,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:59:29,576 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 0d49654bd69ee321ab98d34d9a593b8e 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-06 17:59:29,576 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-06 17:59:29,577 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1479): Online Regions={0d49654bd69ee321ab98d34d9a593b8e=hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e., 1588230740=hbase:meta,,1.1588230740} 2023-04-06 17:59:29,578 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-06 17:59:29,578 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-06 17:59:29,578 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-06 17:59:29,578 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-06 17:59:29,578 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-06 17:59:29,578 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-06 17:59:29,579 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1505): Waiting on 0d49654bd69ee321ab98d34d9a593b8e, 1588230740 2023-04-06 17:59:29,585 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/.tmp/info/8d6ae9a304a54cfdb5edef13cf165357 2023-04-06 17:59:29,590 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8d6ae9a304a54cfdb5edef13cf165357 2023-04-06 17:59:29,668 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-06 17:59:29,779 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1505): Waiting on 0d49654bd69ee321ab98d34d9a593b8e, 1588230740 2023-04-06 17:59:29,980 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1505): Waiting on 0d49654bd69ee321ab98d34d9a593b8e, 1588230740 2023-04-06 17:59:29,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/.tmp/info/681b47acce254685a3c1941f75f54148 2023-04-06 17:59:29,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/.tmp/info/681b47acce254685a3c1941f75f54148 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/info/681b47acce254685a3c1941f75f54148 2023-04-06 17:59:29,995 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/.tmp/rep_barrier/eb317c66b25547d3ae7443fa4f5b18df 2023-04-06 17:59:29,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/info/681b47acce254685a3c1941f75f54148, entries=2, sequenceid=6, filesize=4.8 K 2023-04-06 17:59:29,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 0d49654bd69ee321ab98d34d9a593b8e in 420ms, sequenceid=6, compaction requested=false 2023-04-06 17:59:30,000 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for eb317c66b25547d3ae7443fa4f5b18df 2023-04-06 17:59:30,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/namespace/0d49654bd69ee321ab98d34d9a593b8e/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-06 17:59:30,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:59:30,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 0d49654bd69ee321ab98d34d9a593b8e: 2023-04-06 17:59:30,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680803779493.0d49654bd69ee321ab98d34d9a593b8e. 2023-04-06 17:59:30,180 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 17:59:30,243 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-06 17:59:30,243 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-06 17:59:30,380 DEBUG [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 17:59:30,407 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/.tmp/table/7f45ee0f7f7640639476874323e18e07 2023-04-06 17:59:30,417 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7f45ee0f7f7640639476874323e18e07 2023-04-06 17:59:30,419 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/.tmp/info/8d6ae9a304a54cfdb5edef13cf165357 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/info/8d6ae9a304a54cfdb5edef13cf165357 2023-04-06 17:59:30,423 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8d6ae9a304a54cfdb5edef13cf165357 2023-04-06 17:59:30,423 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/info/8d6ae9a304a54cfdb5edef13cf165357, entries=22, sequenceid=93, filesize=7.1 K 2023-04-06 17:59:30,425 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/.tmp/rep_barrier/eb317c66b25547d3ae7443fa4f5b18df as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/rep_barrier/eb317c66b25547d3ae7443fa4f5b18df 2023-04-06 17:59:30,429 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for eb317c66b25547d3ae7443fa4f5b18df 2023-04-06 17:59:30,429 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/rep_barrier/eb317c66b25547d3ae7443fa4f5b18df, entries=6, sequenceid=93, filesize=5.5 K 2023-04-06 17:59:30,429 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/.tmp/table/7f45ee0f7f7640639476874323e18e07 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/table/7f45ee0f7f7640639476874323e18e07 2023-04-06 17:59:30,433 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7f45ee0f7f7640639476874323e18e07 2023-04-06 17:59:30,433 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/table/7f45ee0f7f7640639476874323e18e07, entries=9, sequenceid=93, filesize=5.5 K 2023-04-06 17:59:30,434 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 856ms, sequenceid=93, compaction requested=false 2023-04-06 17:59:30,445 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-06 17:59:30,445 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-06 17:59:30,447 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-06 17:59:30,447 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-06 17:59:30,447 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-06 17:59:30,581 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,38773,1680803776439; all regions closed. 2023-04-06 17:59:30,593 DEBUG [RS:0;jenkins-hbase20:38773] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/oldWALs 2023-04-06 17:59:30,594 INFO [RS:0;jenkins-hbase20:38773] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C38773%2C1680803776439.meta:.meta(num 1680803779237) 2023-04-06 17:59:30,599 DEBUG [RS:0;jenkins-hbase20:38773] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/oldWALs 2023-04-06 17:59:30,599 INFO [RS:0;jenkins-hbase20:38773] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C38773%2C1680803776439:(num 1680803778328) 2023-04-06 17:59:30,599 DEBUG [RS:0;jenkins-hbase20:38773] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:30,599 INFO [RS:0;jenkins-hbase20:38773] regionserver.LeaseManager(133): Closed leases 2023-04-06 17:59:30,600 INFO [RS:0;jenkins-hbase20:38773] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-06 17:59:30,600 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-06 17:59:30,601 INFO [RS:0;jenkins-hbase20:38773] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:38773 2023-04-06 17:59:30,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 17:59:30,608 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,38773,1680803776439 2023-04-06 17:59:30,608 ERROR [Listener at localhost.localdomain/40169-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@231e8c73 rejected from java.util.concurrent.ThreadPoolExecutor@27210cc8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2523] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 17:59:30,609 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 17:59:30,609 ERROR [Listener at localhost.localdomain/40169-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4a1c913 rejected from java.util.concurrent.ThreadPoolExecutor@27210cc8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2523] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 17:59:30,609 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,38773,1680803776439] 2023-04-06 17:59:30,610 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,38773,1680803776439; numProcessing=1 2023-04-06 17:59:30,611 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,38773,1680803776439 already deleted, retry=false 2023-04-06 17:59:30,611 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,38773,1680803776439 expired; onlineServers=0 2023-04-06 17:59:30,611 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,40911,1680803775368' ***** 2023-04-06 17:59:30,611 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-06 17:59:30,611 DEBUG [M:0;jenkins-hbase20:40911] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4aaba8ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-06 17:59:30,611 INFO [M:0;jenkins-hbase20:40911] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,40911,1680803775368 2023-04-06 17:59:30,611 INFO [M:0;jenkins-hbase20:40911] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,40911,1680803775368; all regions closed. 2023-04-06 17:59:30,611 DEBUG [M:0;jenkins-hbase20:40911] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 17:59:30,611 DEBUG [M:0;jenkins-hbase20:40911] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-06 17:59:30,611 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-06 17:59:30,611 DEBUG [M:0;jenkins-hbase20:40911] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-06 17:59:30,611 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680803777970] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680803777970,5,FailOnTimeoutGroup] 2023-04-06 17:59:30,611 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680803777970] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680803777970,5,FailOnTimeoutGroup] 2023-04-06 17:59:30,612 INFO [M:0;jenkins-hbase20:40911] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-06 17:59:30,612 INFO [M:0;jenkins-hbase20:40911] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-06 17:59:30,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-06 17:59:30,612 INFO [M:0;jenkins-hbase20:40911] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-06 17:59:30,612 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 17:59:30,613 DEBUG [M:0;jenkins-hbase20:40911] master.HMaster(1512): Stopping service threads 2023-04-06 17:59:30,613 INFO [M:0;jenkins-hbase20:40911] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-06 17:59:30,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 17:59:30,613 ERROR [M:0;jenkins-hbase20:40911] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (885058725) connection to localhost.localdomain/127.0.0.1:35475 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-06 17:59:30,614 INFO [M:0;jenkins-hbase20:40911] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-06 17:59:30,614 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-06 17:59:30,614 DEBUG [M:0;jenkins-hbase20:40911] zookeeper.ZKUtil(398): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-06 17:59:30,614 WARN [M:0;jenkins-hbase20:40911] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-06 17:59:30,614 INFO [M:0;jenkins-hbase20:40911] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-06 17:59:30,614 INFO [M:0;jenkins-hbase20:40911] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-06 17:59:30,615 DEBUG [M:0;jenkins-hbase20:40911] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-06 17:59:30,615 INFO [M:0;jenkins-hbase20:40911] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:59:30,615 DEBUG [M:0;jenkins-hbase20:40911] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:59:30,615 DEBUG [M:0;jenkins-hbase20:40911] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-06 17:59:30,615 DEBUG [M:0;jenkins-hbase20:40911] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:59:30,615 INFO [M:0;jenkins-hbase20:40911] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.13 KB heapSize=393.84 KB 2023-04-06 17:59:30,626 INFO [M:0;jenkins-hbase20:40911] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.13 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c18c9e6d2b15491cb8e8885c5477aeb1 2023-04-06 17:59:30,630 INFO [M:0;jenkins-hbase20:40911] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c18c9e6d2b15491cb8e8885c5477aeb1 2023-04-06 17:59:30,631 DEBUG [M:0;jenkins-hbase20:40911] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c18c9e6d2b15491cb8e8885c5477aeb1 as hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c18c9e6d2b15491cb8e8885c5477aeb1 2023-04-06 17:59:30,633 INFO [M:0;jenkins-hbase20:40911] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c18c9e6d2b15491cb8e8885c5477aeb1 2023-04-06 17:59:30,633 INFO [M:0;jenkins-hbase20:40911] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35475/user/jenkins/test-data/07649fd9-79f8-c9ae-4e41-21bf8712c741/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c18c9e6d2b15491cb8e8885c5477aeb1, entries=65, sequenceid=578, filesize=24.7 K 2023-04-06 17:59:30,634 INFO [M:0;jenkins-hbase20:40911] regionserver.HRegion(2948): Finished flush of dataSize ~345.13 KB/353412, heapSize ~393.82 KB/403272, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=578, compaction requested=false 2023-04-06 17:59:30,635 INFO [M:0;jenkins-hbase20:40911] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 17:59:30,635 DEBUG [M:0;jenkins-hbase20:40911] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 17:59:30,637 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-06 17:59:30,637 INFO [M:0;jenkins-hbase20:40911] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-06 17:59:30,638 INFO [M:0;jenkins-hbase20:40911] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:40911 2023-04-06 17:59:30,639 DEBUG [M:0;jenkins-hbase20:40911] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,40911,1680803775368 already deleted, retry=false 2023-04-06 17:59:30,711 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 17:59:30,711 INFO [RS:0;jenkins-hbase20:38773] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,38773,1680803776439; zookeeper connection closed. 2023-04-06 17:59:30,711 ERROR [Listener at localhost.localdomain/40169-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7c63b105 rejected from java.util.concurrent.ThreadPoolExecutor@27210cc8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2523] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 17:59:30,712 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7471ef93] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7471ef93 2023-04-06 17:59:30,712 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): regionserver:38773-0x100876c58980001, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 17:59:30,712 ERROR [Listener at localhost.localdomain/40169-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@8efd104 rejected from java.util.concurrent.ThreadPoolExecutor@27210cc8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2523] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 17:59:30,713 INFO [Listener at localhost.localdomain/40169] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-06 17:59:30,811 INFO [M:0;jenkins-hbase20:40911] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,40911,1680803775368; zookeeper connection closed. 2023-04-06 17:59:30,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 17:59:30,811 ERROR [Listener at localhost.localdomain/40169-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@679abe47 rejected from java.util.concurrent.ThreadPoolExecutor@508331ce[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4222] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 17:59:30,811 DEBUG [Listener at localhost.localdomain/40169-EventThread] zookeeper.ZKWatcher(600): master:40911-0x100876c58980000, quorum=127.0.0.1:49384, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 17:59:30,811 ERROR [Listener at localhost.localdomain/40169-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@30998ee3 rejected from java.util.concurrent.ThreadPoolExecutor@508331ce[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4222] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 17:59:30,813 WARN [Listener at localhost.localdomain/40169] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-06 17:59:30,817 INFO [Listener at localhost.localdomain/40169] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-06 17:59:30,932 WARN [BP-1062686433-148.251.75.209-1680803772339 heartbeating to localhost.localdomain/127.0.0.1:35475] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-06 17:59:30,932 WARN [BP-1062686433-148.251.75.209-1680803772339 heartbeating to localhost.localdomain/127.0.0.1:35475] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1062686433-148.251.75.209-1680803772339 (Datanode Uuid a158d5e1-6698-4857-9005-71ea66cb6362) service to localhost.localdomain/127.0.0.1:35475 2023-04-06 17:59:30,935 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/cluster_00f36afb-ee05-a08a-beea-45e5c08d2a6c/dfs/data/data1/current/BP-1062686433-148.251.75.209-1680803772339] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-06 17:59:30,936 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/58ef28bf-89df-f58f-2607-475d56949b70/cluster_00f36afb-ee05-a08a-beea-45e5c08d2a6c/dfs/data/data2/current/BP-1062686433-148.251.75.209-1680803772339] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-06 17:59:30,968 INFO [Listener at localhost.localdomain/40169] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-06 17:59:31,132 INFO [Listener at localhost.localdomain/40169] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-06 17:59:31,291 INFO [Listener at localhost.localdomain/40169] hbase.HBaseTestingUtility(1293): Minicluster is down